Simply put, timestamps are the part of the log event that tells us when stuff happens. When an event triggers a log, most log formats include a timestamp that tell the user when the event happened. Timestamps are incredibly important since they drive a number of important activities in our product. Most importantly, we use timestamps to filter data in your search results. We also use timestamps to determine when logs are beyond your account’s record retention limit and should be discarded.
The concept is fairly simple, but there are a few things you should know about how we handle timestamps.
Where do we get the timestamp from?
Where the timestamp comes from depends on the type of log data you’re sending through. Timestamps will be prioritized in the order below:
Parsed Log Data
If your data is forwarded by a syslog service, there is a timestamp embedded in the header that we can pull out and use.
If your data is forwarded over HTTP/S, the reception time is used as the timestamp. (Unless the data can be parsed.)
Loggly can display events in either your local or UTC timezone. This will make it easier to read your events, trends, and even dashboards. You won’t need to do mental math to convert the timezone. Furthermore, even if you have servers spread across multiple timezones, all your events will be displayed in local or UTC time. By default, users will see events in local time according to their own browser.
Some people prefer to work in UTC time. For example, you may prefer using a single standard time if you have co-workers in multiple timezones. Each user can change it back to UTC by clicking “Use UTC Time” in their account settings.
When we find that the difference between the event and reception timestamps (aka the drift) is greater than a small value, we will automatically correct your event timestamp. Why do we do this? Well, we’ve found that when the drift is greater than a certain amount, we can usually attribute the lag to the event timestamp not being corrected for timezone changes. This can create issues when we index for search so we’ve worked out a way to correct for that issue.
Let’s walk through an example.
Say the event timestamp in your log is 14:01, but no timezone is included. We note the reception timestamp as 18:01 UTC. Four hours is a huge lag – so we will adjust your event timestamp to 18:01 to correct for what we see as the difference due to timezones.
What if there is “true” drift?
Let’s look at another example where there is true drift, an actual difference between the event and reception timestamp, not due to unrelated influences such as a difference in timezone or an error in server time. Say the event timestamp is 14:01 with a reception timestamp of 18:03. In this case we will assume that the true drift is +00:02 and we will correct the event timestamp to 18:01.
Loggly is designed for near real-time log streaming, so if your timestamps are greater than 24 hours in the past, we will use the time the event was received. Events will not show up in your account if they include timestamps that are older than your retention period, or older than the age of your account.