Support API Paginating Event Retrieval API

Loggly Event Retrieval APIs

Loggly offers two different APIs that you can use for event retrieval.  There is the single-block event retrieval API and the Paginating event retrieval API (currently in beta).  Here’s how they differ:

Single-block event retrieval API Paginating event retrieval API
Operation Asynchronous (non blocking) Synchronous (blocking)
Events Per Retrieval 5000
All Events
Use case Require less than 5,000 events Require more than 5,000 events

Please note:  The Single-block event retrieval API is deprecated and will be End of Life (EOL) on January 1, 2018. We recommend you begin planning to migrate to the Paginating event retrieval API. We encourage you to try the Paginating API and send any feedback to Support.

Paginating Event Retrieval API (Beta)

For performance reasons you might want to paginate the response output. This is because returning the entire data set might be feasible for some queries but prohibitive for others that return a very large amount of data. We now provide a new events Iterator Endpoint to allow paginated output. It is completely synchronous/blocking and does not need a polling / RSID response querying pattern.

 

Events Iterator Endpoint

The base endpoint for the Events Iterator Endpoint is:

https://<SUBDOMAIN>.loggly.com/apiv2/events/iterate
Replace the variable:
 <SUBDOMAIN>: Replace with your Loggly subdomain.

 

Usage:

First, you call the endpoint with query parameters.  The API will respond with the first page (from 50 to 1000 events), if any are found.  If additional pages are available, the response will include a “next” URL that is the last JSON object returned.  Use this URL to retrieve subsequent pages.

 

Note:

  • The results of the initial search are cached.  The cache expires if 10 minutes elapses between calls.  
  • The query parameters cannot be altered or added to while calling the “next” URL.
  • If the count of events is less than the requested page size, then the “next” URL will not be provided on the page.

First call – Create the query:

A set of query parameters must be sent on the initial call to /events/iterate endpoint. These parameters are stored and cached and are automatically applied when the “next” URL is called.

Events Iterator Endpoint Request Parameters
q optional query string, check out the Search Query help. Defaults to “All” events.
from optional Start time (in UTC) for the search. Defaults to “-24h”. (See valid time parameters.)
until optional End time (in UTC) for the search. Defaults to “now”. (See valid time parameters.)
size optional Events returned on each page. Defaults to 50. Maximum size is 1000.
order optional Direction of results returned, either “asc” or “desc”. Defaults to “desc”.

 

Events Iterator Endpoint Response Parameters
events: Each item in the events array will have the following fields:
tags: An array of any tags associated with the event
timestamp: See timestamps to understand how a reference timestamp is derived.
logmsg: The message portion of the log event. (Any headers aren’t included.)
event: Any parsed fields are included.
logtypes: An array of log types that were detected.
raw: The raw event.
unparsed: The portion of the event which was not parsed by our parsers. It will be null if it is not applicable.
id: Loggly’s event ID.
next: URL of the next set of events.

 

Events Iterator Endpoint Usage Example

To see events in pages, start with a request like the examples below. Substitute your subdomain name for <SUBDOMAIN> .

curl --user 'user:password' -XGET 'https://<SUBDOMAIN>.loggly.com/apiv2/events/iterate?q=*&from=-10m&until=now&size=2'

 

Output Example:

curl --user 'user:password' -XGET 'https://<SUBDOMAIN>.loggly.com/apiv2/events/iterate?q=*&from=-10m&until=now&size=1'

 

 

{

“events”: [

{

“raw”: “{\”function\”: \”handle\”, \”publish_success\”: 83, \”timestamp\”: \”17-02-01 00:58:04,852149\”, \”start_time\”: 1485910682.351175, \”args\”: \”\”, \”duration\”: 2501, \”message\”: \”end publish run\”, \”publish_fail\”: 0, \”level\”: \”INFO\”, \”source_duration\”: {\”duration\”: 1, \”end_time\”: 1485910682.351801}, \”feature_duration\”: {\”duration\”: 1427, \”end_time\”: 1485910683.786751}, \”pathname\”: \”/opt/loggly/web/app/alert/management/commands/alerterd.py\”, \”lineno\”: 329, \”cache_duration\”: {\”duration\”: 1, \”end_time\”: 1485910684.821093}, \”action\”: \”alerterd\”, \”update_duration\”: {\”duration\”: 8, \”end_time\”: 1485910684.852121}, \”end_time\”: 1485910684}”,

“logtypes”: [

“json”,

“syslog”

],

“timestamp”: 1485910684852,

“unparsed”: null,

“logmsg”: “{\”function\”: \”handle\”, \”publish_success\”: 83, \”timestamp\”: \”17-02-01 00:58:04,852149\”, \”start_time\”: 1485910682.351175, \”args\”: \”\”, \”duration\”: 2501, \”message\”: \”end publish run\”, \”publish_fail\”: 0, \”level\”: \”INFO\”, \”source_duration\”: {\”duration\”: 1, \”end_time\”: 1485910682.351801}, \”feature_duration\”: {\”duration\”: 1427, \”end_time\”: 1485910683.786751}, \”pathname\”: \”/opt/loggly/web/app/alert/management/commands/alerterd.py\”, \”lineno\”: 329, \”cache_duration\”: {\”duration\”: 1, \”end_time\”: 1485910684.821093}, \”action\”: \”alerterd\”, \”update_duration\”: {\”duration\”: 8, \”end_time\”: 1485910684.852121}, \”end_time\”: 1485910684}”,

“id”: “7ce48bda-e819-11e6-808b-12a1c1f6d2c3”,

“tags”: [],

“event”: {

“syslog”: {

“severity”: “Informational”,

“appName”: “msg”,

“timestamp”: “2017-02-01T00:58:04.852+00:00”,

“facility”: “local use 1”,

“priority”: “142”,

“host”: “127.0.0.1”

},

“json”: {

“function”: “handle”,

“publish_success”: 83,

“level”: “INFO”,

“timestamp”: “17-02-01 00:58:04,852149”,

“start_time”: 1485910682.351175,

“cache_duration”: {

“duration”: 1,

“end_time”: 1485910684.821093

},

“update_duration”: {

“duration”: 8,

“end_time”: 1485910684.852121

},

“duration”: 2501,

“pathname”: “/opt/loggly/web/app/alert/management/commands/alerterd.py”,

“end_time”: 1485910684,

“source_duration”: {

“duration”: 1,

“end_time”: 1485910682.351801

},

“action”: “alerterd”,

“message”: “end publish run”,

“publish_fail”: 0,

“feature_duration”: {

“duration”: 1427,

“end_time”: 1485910683.786751

},

“lineno”: 329

}

}

}

],

“next”: “https://<SUBDOMAIN>.loggly.com/apiv2/events/iterate?next=eea25ee6-0e48-4428-a544-36d6441d132c”

}

Thanks for the feedback! We'll use it to improve our support documentation.