This tutorial will show you how to troubleshoot problems using trend analysis, find the root cause, monitor it on your dashboard, and set an alert. It will guide you through the demo shown in Loggly in 5 Minutes, but on your own account using sample data. The sample data is a smaller set for faster download, so the charts might look slightly different.
Try It With Sample Data
Step 1: Upload Loggly Sample Data
Upload Loggly’s Sample Data, which is a small file containing the events used in this walkthrough. It takes just a single command to upload the data.
Step 2: Search for Your Sample Data
Verify you successfully sent the data to Loggly by searching for all the events you just uploaded using the sample tag.
Step 3: Zoom In On the Events
Zoom in by clicking on the blue column and then dragging with your mouse until it’s evenly distributed across the time series chart. This will make it easier to see trends. You can also Zoom in using the magnifying glass.
Step 4: Save Your Search
You can save this search and time series chart view so you can go back to it later. Call it “Sample Events”.
Step 5: Create a Source Group
Instead of including the tag:sample on every search, create a source group so it will search this tag automatically. Go the Source Setup tab, then click Source Groups. Name the source group “Sample” and enter “sample” as the tag.
Step 6: Plot Maximum Response Time
Let’s imagine we have a problem where results are coming back slow, and we want to troubleshoot and find out why using trend analysis. Search for response time on query calls by selecting the Sample source group, then entering this on the search box.
To plot the maximum response time,
- Click the Charts tab on the screen.
- Select Chart Type as Line.
- Select Value Type Statistics.
- Select the Field as json.querytime_ms and Operator as Maximum.
- Give the Series a Name, Max Response Time
The chart automatically zooms in on the part with data. You can see a few spikes where the responses came back slow.
Step 7: Plot Average Response Time
To compare the maximum to the average response time, click the + icon to add a second series.
- Select Value Type Statistics.
- Select the Field as json.querytime_ms and Operator as Average.
- Give the Series a Name, Avg Response Time
Step 8: Range Search for Slow Responses
To find just the slow events, do a range search for responses over the SLA of 500ms. It must have an upper limit, so make it greater than the maximum response time to show all the slow events.
json.querytime_ms:[500 TO 10000]
Step 9: Filter on Top Failures
To see why they are slow, expand the filter for failures, then click show more to see the top failure code. Clicking on the top failure code will add the filter on that value.
Step 10: See Expanded Event View and Automated Parsing
To learn more about events with this failure code, switch to the event view. Then click on an individual event to expand it out. You will see each field has been automatically parsed out. This is what enables the trend analysis and filters to work on individual fields or facets.
Step 11: Create an Alert
Create an alert so that if responses come back slow in the future, you will receive an email.
- Click on the Alert Bell Button
- Call the alert “Responses Over SLA”, set it so that if happens more than 25 times in 5 minutes.
- Optionally save it as a Saved Search
- You can choose to send this alert to an email or endpoint such as HipChat or Slack. Note: This alert won’t actually activate because you are not sending live data and the saved search is on a custom time range rather than a relative one.
- Check this condition for every 1 minute.
Step 12: Add this chart to dashboard
- Click on Edit Chart icon as shown below. This opens the chart under the Chart tab.
- Give it a name, “Responses over SLA”, then click Save.
- Set Permissions as Team can View by clicking on the Permission icon and then click Save.
- Select existing dashboard from dropdown and add Chart to it.
- Click Save.
Step 13: Create a New Dashboard
- Click the Dashboard tab.
- Click +New Dashboard icon.
- Give it a name, Sample Dashboard
- Click Ok.
Step 14: Create complex charts and add to dashboard
Find existing charts from your account and add to the Dashboard.
Step 15: Send Your Own Data
Go to the Source Setup tab. Send your own log data to Loggly, then setup your own dashboards, alerts, and more!