Dynamic Field Explorer
With Loggly Dynamic Field ExplorerTM, your troubleshooting never begins with a blank search page. Instead, you see a structured summary of all your parsed logs, showing its inherent structure, the identified field names, and the frequency of individual values. It’s like a map of your data, showing you both the most common events and the anomalies, as well as providing a quick and precise way to hone in on specific logs and filter out the “noise”.
As you continue to refine what you are looking for, Dynamic Field Explorer is always updating to provide relevant insights into the data you are viewing right now. The whole process is much faster than starting with a series of trial-and-error searches or having to learn a new search language before you gain any insight.
Common Scenarios Where Field Explorer Will Help
- You are not sure where to start searching and would like a ‘guided search’ experience
- You don’t know the name or syntax of a specific field you want to search on
- You would like an instant summary of all the logs seen during a specific time or a subset of hosts/users, instead of manually checking one by one
- You want to know how frequently a certain event or value shows up in your logs, allowing you to filter out the noise quickly and precisely
- You want to quickly browse your log data to see if any anomalies stand out
Using Dynamic Field Explorer
Dynamic Field ExplorerTM is shown by default in any new tab you create to give you an instant summary of all the field categories that have been identified in the last 10 minutes.
Once you select a field category, you will be shown a list of all the known fields for that category along with a tremendous amount of information and analytics about your log data to help you get to insights faster.
Dynamic Field Explorer really shines is in helping you get to insight or find root of problem even if you don’t know what you are looking for. In fact, many customers find that they are able to get the necessary insight without typing a single search query! This is all possible to the real-time navigable summaries that are created and updated with each and every click or change in your search context.
Note: Dynamic Field Explorer have a limit of maximum 500 fields that could be displayed in the Loggly UI.
Anatomy of Dynamic Field Explorer
- Search fields/values
- If you are looking for a specific field name or value you can use the quick search option by clicking anywhere near the search icon at the top of the Field Explorer
- This will bring up search bar where you can type what you are looking for (will search across all field categories and current selected value list). In the example below the currently selected field is ‘JSON.action’.
- Field actions
- Access to analytical actions you take on the field, such as plotting a trend graph. See below for more details on field actions.
- Field List Panel
- They will be listed in alphabetical order in a hierarchical views (i.e JSON.context.debug will be a tree of 3 levels). Tree views will be by default collapsed. Numeric values are marked with a ‘#’ to distinguish them from categorical fields.
- Protip: The list of fields are dynamically updated after each search context change (search query, time, filters, etc) always giving you an instant summary of the events in you are currently looking at
- Value List & Quick Stats Panel
- All the values for this specific field will be shown ordered by # of times seen in the current search context. Exact count is next to each event
- The header will have the name of selected field, total # of unique values identified, total # of events this field is found in
- Also use it to quickly and precisely filter with a click. You can click more than one
- Slider to customize size of Panel
- Click and hold to drag each panel to the preferred size. Note that there is a minimum size, so if you go beyond that it will automatically snap to collapsed state
- If you just click it will fully collapse
- Quick Recall bar for categories
- Hovering your mouse here will quickly bring back the “cover screen” for field explorer listing out all the field categories available. Use this to quickly switch between categories
DFE also provides common statistics on numeric field values. These include the min, max, mean, standard deviation, and variance of the data. They are visible in the right hand pane after clicking on a numeric field.
You have one-click access to a variety of actions you can take based on a field. There are actions for both categorical and numeric fields.
Categorical Field Actions
- Pie chart: A pie chart of the top values so you can see their relative proportions
- Bar chart: A horizontal bar chart of the top values
- Timeline Chart: Also a timeline representation of the values but now lets you see count over time of the top values
- Copy Values: Copies the values pane as a table to your clipboard so you can easily paste it into a word processor or spreadsheet.
Numeric Field Actions
- Value of: Will give you a visual graph of all the values for that numeric field shown over time so can quickly get insight about its distribution, outliers, etc
- Statistics: Also a timeline representation of the values but now lets you do some quick aggregate calculations (i.e Sum, Average, Max/Min) over time
- Single Value: Similar to statistics but instead of measuring an aggregate over time, it gives you the calculation over the entire search context (i.e if you want to know the MAX value over the entire period).
- Quick Filtering option: Allows you to quickly filter the logs based a numeric range of values
- What type of logs are automatically parsed?
Our Dynamic Fields™ technology automatically parses the most common log types sent by our customers such as Apache, Nginx, Java, Rails, JSON, etc. For a full and most up to date list please visit the Automating Parsing page.
- Can I still filter based on my own custom tags?
Absolutely. If you’ve setup custom tags for your data, you’ll find them (along with other special attributes) under “Other” along with the logtype field
- What happens with events that are not parsed?
Dynamic Field Explorer will only provide summary data for events with parsed fields. For other events, you can continue to leverage the other robust search, navigation, and graphing capabilities to quickly find the data you are looking for.
- Can I define rules to custom parse parts of my data you may not be automatically parsing?
Today, the best way to do such is to use readily available 3rd party tools to translate custom log formats into JSON and then send them to Loggly (find out more at bottom of Automating Parsing page). We are also working on ways to streamline this definition within the Loggly console so would love to hear your feedback if this is an area of interest.
- I’ve sent Numeric fields over in my custom JSON but don’t see them. How can I search based on those fields?
Numeric fields are currently not exposed through the Dynamic Field Explorer interface, but support for them is coming very soon. In the interim, you can continue to search across numeric fields and ranges using the search bar.
- Why do I only see a subset of values for a certain field (i.e. I know more values exist)?
Remember that Dynamic Field Explorer is an automatically generated summary of the events that exist in your current search context. Most likely reasons are that these are the only values seen during this time range or subset of data. Easiest way to double check is just do a blank search over a wider timerange and you will see all the fields and values that have been detected.
- Why are some fields not showing up in the Field Explorer?
Double check that your field does not have numeric values as that is not supported yet but coming soon. If you are not seeing fields that you expect to see, please contact Loggly support and we can investigate.