What are Derived Fields?

Loggly derived fields let you define custom parsing rules for log events. With derived fields, you can add structure to logs, create new fields, and index events in ways that matter to you. It’s a way to extend Loggly’s powerful built-in parsing engine with your own unique parsing rules.

Like normal parsed fields, derived fields are parsed as Loggly receives the log message. This gives you instant access to the information in your logs as soon as you save your new rule. This also opens up your previously unparsed logs to other Loggly features such as advanced filtering, aggregation, calculations, graphing, and alerting. You can also add additional parsing rules to compliment our automatic ones.

Derived fields make it easy to extract information from unparsed fields, or even parse formats that aren’t currently recognized by Loggly. The derived fields are automatically indexed and appended to the original log event, allowing you to easily index and search the event through the Loggly Dynamic Field Explorer™.

 

Creating Derived Fields

The fastest way to get started with derived fields is by clicking the “Create Derived Fields” button when viewing a log event. This will take you to the Rule Manager, where you can define your new derived fields rule.

derived-fields-start-rule

You can also create a new rule by clicking the Derived Fields button at the top of the screen.

derived-fields-start-rule

Derived Field Rules

Derived field rules contain the logic necessary to extract your new fields from incoming log events. Almost any kind of log event can be broken up into fields that can indexed by Loggly. There are currently four supported types of rules: key-value pairs, anchors, regular expressions (RegEx), and insert tags.

 

Derived Field Rule Types
Type Description Example
Key-Value Key-value fields consist of a field name and value separated by a delimiter. Unparsed status_code=700
Parsed status_code: 700
Anchor Anchor fields extract a value by matching against surrounding text. In this example, the preceding anchor is “[client ” and the following anchor is “]”. Unparsed [client 127.0.0.1]
Parsed host: 127.0.0.1
RegEx RegEx rules extract log data using regular expressions. This example uses the following RegEx to extract the “until” date: ^.*until=(d{4}-d{2}-d{2}).*$ Unparsed from=2015-10-01T18%3A34%3A03.390ZJW235&until=2015-10-02T18%3A34%3A03.390Z
Parsed until: 2015-10-02
Insert Tag Insert tag rules will insert a tag into a matching event. You can even use multiple rules to mark different events with a common tag. This example uses the following RegEx to tag events containing “Exception”: ^.*Exception.*$ Unparsed “Sqsqueue_Exception”: “No messages received”
Parsed Tag: Exception

 

While you can only have one rule type active at a time, each rule provides a number of options for greater control. You can also run multiple derived field rules against an event.

 

Key-Value Rule Definition

Key-value rules identify sets of two related data points by searching for a separator (a character that links a key and its value) and a delimiter (a character that separates two sets).

 

derived-fields-key-value

Key-value rules are particularly powerful since a single rule definition can create multiple fields. This lets you easily extract several fields without having to define multiple rules. Note that both keys and values are stored in Loggly as strings.

 

Key-Value Rule Configuration
Parameter Description Example
Separator A character that separates the key from the value. user=sim27
Delimiter A character used after a key-value pair that signifies the end of the pair. key=value, key2=value2
Strip surrounding quotes When checked, any single quotes, double quotes, or backticks surrounding the key or value are removed. user=sim27
Do not extract key-value pair if key is: The name of a particular key that you don’t want to parse. user
Extract Field From Lets you select a previously parsed field to parse further. syslog.appName

Anchor Rule Definition

Anchor rules extract values from events based on their context. With an anchor rule, you define the patterns that precede and follow the desired value. For instance, the following example shows how to extract the IP image by searching for [client_ip and ]. Note that unlike a key-value rule, an anchor rule will only extract the first match that it finds.

 

derived-fields-anchor

 

Anchor Rule Configuration
Parameter Description Example
Field preceded by: The string that precedes the field of interest [client_ip 127.0.0.1]
Field followed by: The string that follows the field of interest [client_ip 127.0.0.1]
Extract Field From: Lets you select a previously parsed field to parse further. syslog.appName

 

RegEx Rule Definition

RegEx rules let you use regular expressions to match against your logs. Regex rules must match the entire event by starting with ^ and ending with $. Like an anchor rule, a RegEx rule will extract the first match that it finds.

 

RegEx Rule Configuration
Parameter Description Example
Event matches RegEx: The regular expression that you want to match, including capture groups. ^.*until=(d{4}-d{2}-d{2}).*$
from=2015-10-01T18%3A34%3A03.390ZJW235&until=2015-10-02T18%3A34%3A03.390Z
Extract Field From: Lets you select a previously parsed field to parse further. syslog.appName

 

Insert Tag Rule Definition

Insert tag rules are somewhat different than key-value, anchor, or RegEx rules. Rather than extracting data from an event, an insert tag rule lets you tag the event based on its contents. The rule matches events using a regular expression, then adds the specified tag to the event. Like the RegEx rule, the regular expression used in an insert tag rule must match the entire event by starting with ^ and ending with $.

 

RegEx Rule Configuration
Parameter Description Example
Tags to insert Name of the tag that you want to insert. access_log
Event matches RegEx The regular expression that you want to match, including capture groups. ^apache-access$
Look in: Lets you select a field to match the RegEx against. json.basicinfo

 

Limiting Rule Scope

You can limit the scope of all derived field rules by specifying the source or contents of the event. Loggly recommends limiting your rules as much as possible to reduce the chances of extracting unnecessary or unwanted data. For instance, you might want to extract the IP address from logs originating from your load balancer, but you probably don’t want the same rule to apply to logs originating from a test or staging server.

 

You can limit your rule’s scope using four parameters: hosts, applications, tags, and events containing text. If you entered the derived field editor through a log event, these fields will auto-populate with any parameters available in the event.

 

Limit Rule Scope Configuration
Parameter Description
Host The name(s) of one or more hosts that the logs will originate from. Note that host names are only available with syslog events.
Application The name of the application(s) or service(s) that the logs will originate from. Note that host names are only available with syslog events.
Tag The name(s) of a Loggly tag that will be applied to the log event.
Events containing text As an alternative to the above fields, you can filter on events if they contain specific text.

 

Each parameter lets you enter one or more values for filtering. As you type, Loggly provides suggestions based on previous events. You can click on the suggestion to replace the current text with the suggested text. Clicking out of the text box or pressing enter will also convert the typed value into a valid target.

derived-fields-limit-scope

You can also use wildcards to expand the range of your derived fields. For example, the following scope limits the scope of the rule to logs originating from Docker or from any daemon running on the system. Note how Loggly displays “Multiple Applications” instead of “2 Applications,” which it would have displayed if we specified two distinct values. Note that wildcards can only be used for hosts, applications, and tags.

 

derived-fields-limit-scope-rule

Testing Your Rule

Before you can save your rule, you need to test it against existing log events to verify that it will correctly filter new events. The test runs against a set of sample logs displayed beneath the rule editor. If you want to specify your own set of test logs, click the “Modify Search” link to open search controls. Enter your parameters, then press “Search” to populate the set with a new list of events.

 

derived-fields-test-rule

After the new set is loaded, press “Test Rule on Events” to test your rule. If the test was successful and you’re satisfied with the results, you can press “Continue” to move to the next step.

 

derived-fields-test-rule

Saving Your Rule

The final step is to save your new rule. Here, you can name your field rule, as well as specify the name and type of the field that contains the extracted data. For key-value rules, the field name is automatically set to the extracted key and the data type is always string. This option is unavailable for insert tag rules, since no new fields are created. When you’re finished, press the “Save Rule” button.

 

derived-fields-save-rule

Now that you’ve saved your rule, Loggly will inspect and automatically parse any new logs that match the requirements for your rule. When an incoming log matches the rule, the new fields will appear in the Dynamic Field Explorer. The Dynamic Field Explorer will also contain a new Derived Fields category. Clicking it allows you to browse through all of your derived fields. You can also search on your derived fields by entering “derived.<fieldname>” in the search bar. Now you can create summaries and visualizations using your custom logs!

 

Frequently Asked Questions About Derived Fields

1. How do I force my derived fields to be numeric?

When saving a RegEx or anchor rule, there will be an option to define the field as a double, long, or timestamp. At the moment, key-value rules can only be stored as strings.

2. How do I force a derived field to be treated as the searchable timestamp?

Select ‘timestamp’ from the dropdown next to the desired field when saving your rule. We will then automatically parse out the timestamp from the field as long as it is in one of the supported formats:

yyyy-MM-dd’T’HH:mm:ss
yyyy-MM-dd’T’HH:mm:ssZ
yyyy-MM-dd’T’HH:mm:ss.SSS
yyyy-MM-dd’T’HH:mm:ss.SSSZ
yyyy-MM-dd HH:mm:ss,SSS
yyyy-MM-dd HH:mm:ss,SS
yyyy-MM-dd HH:mm:ss,SSZ
yyyy-MM-dd HH:mm:ss,SSSZ
yyyy-MM-dd HH:mm:ss,SS z
yyyy-MM-dd HH:mm:ss,SSS z
yyyy-MM-dd’T’HH:mm:ss.SSSSSS
yyyy-MM-dd’T’HH:mm:ss.SSSSSSZ
yyyy-MM-dd’T’HH:mm:ss.SSSSSSS
yyyy-MM-dd’T’HH:mm:ss.SSSSSSSZ

Note: For formats yyyy-MM-dd HH:mm:ss,SS z and yyyy-MM-dd HH:mm:ss,SSS z , the timezone must be specified as GMT, rather than UTC.

3. Do you recommend any tools for learning about RegExes?

There are many great resources online for learning and experimenting with regular expressions. We recommend regexr.com, since it offers a live testing sandbox as well as cheat sheets.

4. How do I search for the events on which a derived rule was applied?

When Loggly receives a matching event as per the derived rule then we add the logtype automatically to those events in this format:

log type: name_of_the_derived_rule

You could search for the events on which a derived rule was applied by searching as:

logtype:”name_of_the_derived_rule”


 

Thanks for the feedback! We'll use it to improve our support documentation.


Top