Amazon CloudTrail Logs

Loggly provides the ability to read your AWS CloudTrail logs directly from your AWS S3 bucket. Amazon CloudTrail support is built into the Loggly platform, giving you the ability to search, analyze, and alert on AWS CloudTrail log data.

What Can I Do With AWS Cloudtrail Logs?

Cloudtrail logs keep a record of all AWS API calls and help you answer key security and compliance questions. Amazon describes this in detail in their white paper called Security at Scale: Logging in AWS. Here are some common questions you can answer:

● What actions did a user take over a given period of time?
● For a given resource, which AWS user has taken actions on it over a given time period?
● What is the source IP address of a given activity?
● Which user activities failed due to inadequate permissions?
● Which user changed the settings of a security group and when did the change occur?
● When was a particular Elastic IP (dis)associated with a network interface?
● Which user launched or terminated an EC2 instance?

Configuring Cloudtrail Logs with Loggly

Loggly reads AWS CloudTrail logs directly from your AWS S3 bucket. Here’s how to give us permission and configure Loggly to read them.

Step 1: Log into your AWS console

If you don’t already have one, you’ll have to create an Amazon account.

Step 2: Subscribe to CloudTrail

From your AWS console, choose “CloudTrail” from the Deployment & Management section.

AWS CloudTrail Setup

The subscription to CloudTrail itself is free, the only charge is for storage on the S3 bucket. Provide a name for the new S3 bucket that will hold the CloudTrail logs. (Remember the name you provide here, you’ll need to reference it a few times during setup.)

AWS CloudTrail Logs

Step 3: Provide permission to Loggly to read from the bucket

Loggly will need permission to pull the CloudTrail log data from your S3 bucket. The easiest way to accomplish this is by creating a new IAM user on your account. The new user will have only have permission to read from the S3 bucket.

Go back to your AWS dashboard and select “IAM” from the Deployment & Management section

AWS CloudTrail Dashboard

From your IAM dashboard, choose Users from the left-hand menu. Then, create a new user & make sure to download the credentials. (You’ll need to provide these to Loggly in Step 4.)

Create New AWS User

Download Cloudtrail User Credentials

Once the user is created, select the user from your user list. Under the “Permissions” tab, choose “Attach User Policy”.

Attach CloudTrail User Policy

In order to configure the permissions for the new user, select “Policy Generator”. Loggly will need access to list the contents of the bucket & to get objects within the bucket.

Manage CloudTrail User Permissions

Grant List Bucket Permissions

  • Effect: Allow
  • AWS Service: Amazon S3
  • Action: List Bucket

Select Service and ListBucket Action
Enter the name of the bucket you created in Step 2. You’ll need to use the Amazon Resource Name format. e.g. if the name of the bucket you set up is “cloudtrail-bucket”, enter this:

 arn:aws:s3:::cloudtrail-bucket

Enter Bucket ARN

Grant Get Object Bucket Permissions

  • Effect: Allow
  • AWS Service: Amazon S3
  • Action: Get Object

This time, the ARN needs to point to the specific location of your CloudTrail logs. In most cases, this will just be something like:

arn:aws:s3:::cloudtrail-bucket/*

If you selected a file prefix during CloudTrail bucket setup, be sure to specify it here:

arn:aws:s3:::cloudtrail-bucket/prefix/*

Add GetObject Permission

Double check that you’ve added the necessary permissions, click “Continue”.

Complete Permissions

Name the policy, e.g. loggly-cloudtrail-policy, and click “Apply Policy”.

Name and Apply Policy

Step 4: Establish your new S3 bucket with Loggly

Now we come back to Loggly. Once you’ve set up CloudTrail and an IAM user, you’ll need to give us that information so we can read from the bucket. Only account owners and account administrators can set up CloudTrail within Loggly. If that’s not you, contact the account owner before you can continue.

If you are the account owner or admin (lucky you!) go to the account page in Loggly and select AWS CloudTrail.

AWS CloudTrail Loggly

Enter each of these fields into the input boxes then click Save:

  • S3 Bucket Name – the name of the bucket you entered in step 2
  • Prefix (optional) – a key prefix or directory to store the logs in, must include a / at the end
  • Access Key ID – the access key ID you received in step 3
  • Secret Access Key – the secret access key you received in step 3

Step 5: We pull logs from your S3 bucket.

That’s all you need to do. Once we verify access to your S3 bucket, we’ll stream the log data directly to Loggly.

After you first set-up an S3 bucket it may take a few hours for the configuration to complete.

Head over to the Loggly Search page and perform a search for

 logtype:cloudtrail

CloudTrail Logtype

You’ll find all of your CloudTrail logs are fully parsed & ready to be analyzed. Look for the logtype “cloudtrail” and all other fields will be prepended by “json.”, e.g. json.sourceIPAddress. Here are a few example searches:

Find the top events within your CloudTrail logs, but don’t include the Describe events:

logtype:cloudtrail NOT json.eventName:describe

Find who is using Root permissions the most often:

logtype:cloudtrail  json.userIdentity.type:"Root"

And then look at the left-hand panel to find which sourceIPAddress is generating the most requests.

Troubleshooting AWS CloudTrail Logging

Already configured Cloudtrail, but don’t see events yet?
When you first set up Cloudtrail, we will ingest all the Cloudtrail logs that we find in your bucket. Depending on the volume of your logs, it may take some time to start seeing them (especially most recent events). Please feel free to contact our support team if your logs do not appear within 24 hours.

Why is there a spike of Cloudtrail events in the first few days?
By default, Loggly will mark each ingested event with the timestamp found in the Cloudtrail information. However, if the timestamp is more than 7 days in the past we will mark the event with the ingestion time instead. This will only affect what day/time that event gets attributed to in the UI but does not modify the actual timestamp data in the raw event.

Please see these related links for: AWS Config Logging, AWS S3 Logs, and AWS SNS messages.

Thanks for the feedback! We'll use it to improve our support documentation.


Top