Search Query Language

Loggly’s search query language is based on Apache Lucene. Review the following sections below to start putting your data to work:

The Basics

Your First Search

Leave the search box blank and click Search. This way, you’ll see a full picture of what log data exists within Loggly.

Terms

Simple strings, such as “foo” and “867”, phrases, such as “local use 6″ can be entered directly into the search box. Terms can be combined together with Boolean operators to form a more complex query.

Operators

When using more than one search term in your query, you’ll need to join them together with an operator. Loggly uses the Boolean operators ‘AND’ ‘OR’ and ‘NOT’ as well as a final operator ‘TO’.

Operator Description
AND The intersection of terms. Both terms must exist in the event.
OR The union of terms. Either term may exist in the event.
NOT This term must not exist in the event.
TO Used when searching for ranges of values. A query term using the ‘TO’ operator must take the form of <field_name>:[low# TO high#]. The operator ‘TO’ can be used with numeric values as well as words that have lowercase letters, i.e., syslog.severity:[informational TO warning]. A field name must be supplied.

This example would find log events that have both a hostname of frontend01.example.com and a logtype of apache.

syslog.host:frontend01 AND logtype:apache

This example would find log events that have either a hostname of frontend01.example.com or frontend02.example.com

syslog.host:frontend01 OR syslog.host:frontend02

This example would find log events that have frontend01.example.com, but not frontend02.example.com

syslog.host:frontend01 NOT syslog.host:frontend02

This example would find response times that are between 100ms and 500ms

json.responseTime:[100 TO 500]

This example would find response times that are greater than 500ms

json.responseTime:[500 TO *]

As an alternative to the Boolean operators above, you can also use the ‘+’ and ‘-‘ sign in place of ‘AND’ and ‘NOT’ respectively. These symbols must be used before the query terms they reference without spaces. For example, these search queries are equivalent:

"user list" +micheletto -rufio
"user list" AND micheletto NOT rufio
"user list" micheletto NOT rufio

A space is interpreted as the ‘AND’ operator by our engine. Hence the full text search “error 404″ is interpreted as “error AND 404″. We don’t recommend using this shortcut for any but the shortest search queries – it can quickly become hard to decipher search terms separated by only a space in long queries.

Grouping

Query terms can be grouped using parentheses. Terms in parentheses are processed first. This example will find events with a term “user list” that includes either micheletto or rufio.

"user list" (micheletto OR rufio)

Field Names

The full topic of structured vs unstructured data is covered in more detail in the Log Types section, but we will talk about them in the context of search here.

All logs, both structured and unstructured, will have a full-text index. Meaning, every string within your data is searchable.

Additionally, supported log types are parsed, which yields field names and field values from your logs. This means you can search for particular strings or ranges of numerical field values within specific field names. Each field name is prepended with the log type: apache, json, syslog, etc. For example, if you have apache logs you could perform search queries such as:

apache.status:[500 TO 599]
apache.referer:google
syslog.host:frontend*

If the data includes nested JSON, the field name should be given with dot notation:

json.sport.baseball:Giants

Using operators and filters strategically can enhance the specificity of your search results.

You can also do an exact match search on field values using the elastic search syntax of ._facet. This can be useful if there are special characters or others that are important to getting accurate results. This only works if the length is less than 100 characters. Otherwise you can only do the tokenized search.

syslog.host._facet:"frontend01.loggly.com"

Using Wildcards

The wildcard symbols can be used to bring up results with partial search terms entered. Wildcards can only be used standalone or when trailing a partial term.

Operator Description
* Will match more than one character.
? Will match just one character.

Valid use of wildcard expansion:

json.level:*
*
json.level:er*
json.level:erro?

Invalid use of wildcard expansion:

*.level:info
*nfo
json.level:*er

Tokenization

When Loggly indexes your content, we do a full text index, which means that every single string and numeric value is available to search across. Each string and numeric value we find is considered a “token”. We also break tokens when there is a change in capitalization. For example, if we spot a string like:

frontEnd01.example.com

It’s actually broken apart into tokens. We’ll insert a space for any non-alphanumeric character and we’ll split apart any numeric from the alphabetic characters. Any of these searches will match that value:

frontEnd
frontEnd01
End
01
example
com
"frontEnd01.example.com"
"frontEnd01 example com"
front*

Regular Expression Searches

To further power your searches, try using regular expressions to match different patterns. Include your regex between forward slashes “/”. For example, to search for either wall or ball, the following will work:

/[wb]all/

To search for more than four 0’s in a row:

/0{4,}/

Regular expressions only work on single tokens when run on a full text search. You cannot do a regex on a single term that has been split into several tokens due to special characters or other rules. For example, this won’t work because it’s been tokenized, even if we escape the special characters:

/HTTP/1.0/

Instead, consider doing a search on ordered tokens by quoting the search term.

"HTTP/1.0"

You can also do an exact match search using elastic search’s ._facet notation. This is useful if you want to search without tokenization or if special characters are important in your search.

syslog.host._facet:"frontend01.loggly.com"

For further details on the regular expression syntax we support, please review Lucene’s Regular Expression class. WARNING: The Regular Expression syntax is experimental and might change in incompatible ways in our next release.

Non alpha-numeric symbols in Search

Only alpha-numeric characters will be matched within log events. Special characters + – & || ! ( ) { } [ ] ^ ” ~ * ? : \ _ cannot be matched and should not be included as part of the search query. If you do include special characters within your search query, be sure to use quotes around the term. For example, an unquoted colon will cause an error if it’s not preceded by a valid field name.
These two searches will show equivalent results:

 "http://yourwebsite.com:9000"
"http yourwebsite com 9000"

This search will result in an invalid query since the colon is not quoted:

 http://yourwebsite.com:9000

Case sensitivity

Loggly is case-insensitive to all terms in a search query EXCEPT in two cases:

  • The ‘TO‘ operator
  • Field names, including tag and logtype

Time Ranges in Search

Time ranges can be relative or absolute. Relative time ranges are pegged to the current time. Quick picks for some relative ranges are available. To supply an absolute time range or a custom relative range, choose “Custom time range” from the date selector to see the options:
custom_time_range

Custom Time Range Syntax

Type Description Example
Absolute Dates yyyy-MM-dd HH:mm:ss.SSS 2013-08-11 00:00:00.000
yyyy-MM-ddTHH:mm:ss.SSSZ 2013-08-13T18:45:00.000Z
Relative Dates now now
Last 5 weeks -5w
Last 12 hours -12h
Last 5 minutes -5m
Last 30 seconds -30s

Logs that are beyond their retention period are not included in search results.

Source Groups

One source group can be provided for a search. By narrowing down the search before you even start, you can be assured that your search will return relevant results. For example, I may want to search just my staging logs for an event that happened on my application servers. If I include a set of hosts called “staging*” and including a tag called “python”, I can recall this Source Group when necessary. Source Groups are optional groupings, but very useful for segmenting large amounts of disparate data. Source Groups are fully explained later in this document.

Filters

Filters are another method of narrowing down your search query. The “Filter by Field” section on the left hand side of your search screen may show a list of fields. Each of these field names, when clicked, will show the list of top values beneath. Click “Show All” to see all of the top values for each field. The Filter by Field section is explained later in this document.

Advanced Queries

Loggly also supports several types of Lucene or Elastic Search based queries.

Proximity match allows you to search for tokens within a certain distance of each other. For example, this will find the tokens user and root within a distance of 4 tokens.

“user root”~4

Fuzzy match allows you to search for a token within a character edit distance of 2. For example, this will match fuzzy or fuzzy1.

fuzz~

Search Tips & Tricks

search_tips

Here are some things to keep in mind while searching if you’re not seeing the results you expect:

1. Clear Source Groups

Change your Source Group to “All Sources”

2. Special characters should be quoted or avoided

Special characters: + – & || ! ( ) { } [ ] ^ ” ~ * ? : \

Many of these characters are used by Loggly’s query language, so shouldn’t be part of your query term.

3. Expand your time range

Check your time range, you may not have noticed that you’re searching for seconds instead of hours.

4. Remove filters and terms

Check for the pills under the search box. Reduce the number of terms you’ve included.

5. Check your spelling

Watch out for typos… both in your log event and in the query.

6. Don’t use partial strings without a wildcard

Here’s an example log message:

Timer-2 S3Stats.processStatEvents INFO: Processed stats for Stats-6297-138

That event will be indexed by the following strings:

timer 2 S 3 stats process stat events info processed stats for stats 6297 138

Because of that, the following terms will match the event:

timer
process
event*

These terms will not match the event:

event
time
Thanks for the feedback! We'll use it to improve our support documentation.


Top