This document covers basic questions about Loggly's logging service. If you have a question that doesn't appear here, please visit the forum.
Loggly has many different protocols that are accepted, including:
If you have a device or application that can connect to us via one of those protocols, you can send us log data. There isn't a log forwarder or agent that you need to install locally.
As one of the collection methods, Loggly runs a syslog based logging service. In the most simple terms, if you point your syslog server at logs.loggly.com, we'll catch and eat those logs and put them in your account for you. There are instructions for configuring syslog, syslog-ng, rsyslog, Splunk, and other systems to send us logs. Check the logging configurations below for more info on how to get your logs to us.
An input is a port, on Loggly, which accepts syslog or HTTP/S formatted event data. Inputs are used to segment log data within Loggly. Think of them as logical buckets.
A device is a machine, host, or source which sends data to a Loggly input. You can view the devices sending data to an input from the input detail page on your account. Devices are used for access control purposes, and conducting searches for events sent from them. Each input has a set of devices assigned that are allowed to send data to that input.
Hoover is the beaver who serves as Loggly's mascot! Hoover was designed by Andre Jolicoeur, and was named after the mighty Hoover Dam. Hoover is Loggly's main marketer, and finds all sorts of interesting topics to tweet about. Hoover and his image has been trademarked, so he can be considered a 'made beaver'.
The Loggly proxy is a specialized syslog server which listens on multiple ports for your syslog or HTTP/S POSTed data. Loggly runs multiple proxy servers, which can be accessed by using the name logs.loggly.com.
The Loggly Shell is the primary interface for searching log events. While the shell appears to be a old school terminal window, it's actually an advanced AJAX-base application which can display graphics, text results, and more.
Each customer on Loggly has a subdomain which is used to access the account. An example of an account on Loggly is: mylogs. That account can be accessed by using the URL: http://mylogs.loggly.com. When you sign up for a Loggly account, you'll need to provide a subdomain that you'll then use to access your account. Subdomains cannot be changed.
Loggly can operate Syslog in two modes. The normal mode is where we index exactly what we receive via syslog. The other mode, the “stripping” mode, is where we eliminate the syslog headers before we index and store the data. Let me explain with some examples: Assume you are sending your data via straight syslog. You configured your device to log via syslog to Loggly. In this case, you would like to preserve all the syslog headers that the original syslog added to the event. An example of an event that you log and is going to be indexed in Loggly looks as follows:
Feb 16 15:33:22 app-1 collectd: uc_insert: Value too old: name = i-d55fbbbe/df/df-dev-shm; value time = 1266363202; last cache update = 1266363202;
This is the normal case and you would use the input type of “Syslog UDP” or “Syslog TCP”. If you configured “Syslog UDP Header Stripping” or the TCP analogous entry, you would end up with the following event being indexed in Loggly (note how the header got stripped off):
uc_insert: Value too old: name = i-d55fbbbe/df/df-dev-shm; value time = 1266363202; last cache update = 1266363202;
The case where you should be using the “Syslog UDP Header Stripping” is where you are tunneling events through syslog. For example, if you are reading from a file with SyslogNG and forward the content of the file to Loggly, you want to enable the stripping option. This is the case, for example when you monitor Apache logs. The following show what happens when you don't enable header stripping:
Feb 16 15:33:43 app-1 apache: 10.214.46.230 - - [16/Feb/2010:15:33:42 -0800] "GET /wp-content/themes/optimize/includes/js/reflectionXXX.js HTTP/1.1" 404 2326 "http://www.loggly.org/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10" KgR2G66BhkYAAEcPAwYAAAAD 228199
Notice how you end up with an additional syslog header. You don't want that. That is why you use the “Syslog UDP Header Stripping option to preserve only the original part of your log entry:
10.214.46.230 - - [16/Feb/2010:15:33:42 -0800] "GET /wp-content/themes/optimize/includes/js/reflectionXXX.js HTTP/1.1" 404 2326 "http://www.loggly.org/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10" KgR2G66BhkYAAEcPAwYAAAAD 228199
The short answer is that you should use TCP. The difference between UDP and TCP is that UDP is not reliable. Each packet is sent from your computer to our proxy server. If UDP is used, it could happen that the packet gets lost and therefore your log entry. In the case of TCP, it is guaranteed that the packet makes it to the proxy. If it should get dropped on the way, TCP will resend the packet and make sure that your log record arrives at Loggly. In terms of security, you are safer using TCP as well. It is not easy to spoof TCP messages, whereas UDP messages are incredibly simple to spoof.
You can think of input names as a way of more finely slicing up the data within all the data you send us. You should use names that clearly identify what kind of data is being handled by the input. We do some processing of the names as we index them, which allows you to search for data across multiple related inputs very easily. To get the full details on this, take a look at the Input Names page.
When you create an input, it is automatically in discovery mode, which means that you can send logs to that port from any machine you like. After 15 minutes, the input will fall out of discovery mode. It basically blocks any connection to that port. Except for the machines that have been sending data to that port during the period that the input was in discovery mode. This helps prevent that other people can send data to your inputs.
At any point in time, you can click on the “discover” slider to put an input back into discovery mode for 15 minutes. The machines that are allowed to send data to an input are called “devices”. You can see a list of devices that are allowed to send data to an input on the input detail page.
Short answer: No. Long answer: It's possible to send historic data, however, Loggly uses the time at which we receive the event as the timestamp for the event. While it is possible to upload a file using curl on an HTTP(S) input or poll an older file using a syslog daemon the fact that the timestamp is recent almost defeats the purpose. If you do send historical data searching is best accomplished by using an epoch timestamp as a field within your JSON logs.
Make sure on a delete request to send the parameters as GET parameters, not POSTs:
curl -X DELETE -u <user>:<pass> http://<subdomain>.loggly.com/api/devices/?ip=10.0.0.3&input_id=879
curl -X DELETE -u <user>:<pass> http://<subdomain>.loggly.com/api/devices/10.0.0.3/
Make sure to check the API Documentation for how to access and manipulate the objects in the system.