Loggly stores your logs in a large scale search engine hosted on the Internet. The amount of time we store your logs in our search engine index is called the 'index retention time' which you can set from your pricing tab under your account. Once events in an account reach an age that is older than the max index retention time for your account, the events are removed from the index.
Loggly provides a way to archive logs older than your account's index retention time by writing them to your own S3 bucket. We'll create folders that are named after your Loggly input ID. Logs in your bucket are kept forever, or until you remove them, so you'll always have a copy handy if you need them.
You can create an Amazon S3 bucket, authorize us to write to it, and give us the bucket name, and then we'll write your logs into that bucket from then on.
To set up a bucket for writing, head on over to the Amazon S3 dashboard. If necessary, make yourself a new bucket by using the create bucket button. Ensure your bucketname is valid and follows the S3 naming rules. Example: "new-s3-loggly-bucket". There will be a logging option on the create button modal, but you can just ignore that as it doesn't have anything to do with Loggly.
Once you have the bucket created:
Back over on Loggly, go to your account page (yoursubdomain.loggly.com/account/archiving/) and enter the name of your bucket in the form. Click submit, we'll make sure we can write to your bucket and will start flinging logs into your S3 bucket as we get them.
NOTE: Only account owners have permission to set up archiving.
Events are written to your S3 bucket in a .part format. The .part files are temporary files that Loggly needs in order to properly merge your data. Please don't edit or open these .part files. The .part files will be processed approximately every two hours into the format specified on your archiving page (more on this just below). This process is similar to .rar files being combined when ALL rar files are present.
There are three options for the format you would like your logs stored as:
Note: It may take upwards of an hour before you start seeing logs in your bucket.
There are several clients available for browsing your S3 buckets. If you are using OSX, check out S3Hub. At $2.99 on the App Store, it's a heck of a deal.
Another great tool that we use at Loggly is the S3cmd CLI. If you're using Ubuntu it's normally in the default repositories (sudo apt-get install s3cmd). S3cmd can also be found on github!