Support Logging setup Amazon CloudWatch metrics

Send Amazon CloudWatch Metrics to Loggly

Metrics are stats relating to the performance of your systems. You can push your Amazon CloudWatch metrics to Loggly using our AWS Lambda Script. Please note these instructions are for Cloudwatch Metrics, which are different from CloudWatch logs.

AWS Setup

For this step you need to install the AWS Command Line Interface (AWS CLI). If you don’t have the AWS CLI installed, click here to learn more about its installation and usage.

NOTE: Always use the latest version of AWS CLI to get all the updated features. You can update the AWS CLI using the command below:

pip install --upgrade awscli

If you are new to AWS, follow this link to learn how to get AWS Access Key, Access ID and Region information.

1. Encrypt Loggly Customer Token

Please make sure that the AWS CLI is configured and is using the correct AWS Access Key ID, AWS Secret Access Key and Default region name using ‘aws configure’ command.

aws configure

Create a KMS key using the steps mentioned in the link with alias name: logglyCustomerToken
Encrypt the Loggly Customer Token using the AWS CLI. Please retain the quotes around the TOKEN below.

aws kms encrypt --key-id alias/logglyCustomerToken --plaintext "TOKEN"

Replace:

Sample Output:

You should see something like the output below. If you are getting any error,  please make sure the user has access to the KMS key.

sample output

2. Get the Lambda Code

Clone the git repo cloudwatch-metrics-to-loggly

git clone https://github.com/psquickitjayant/cloudwatch-metrics-to-loggly.git
cd cloudwatch-metrics-to-loggly

Install the required npm packages.

npm install

Edit index.js to set up an encrypted customer token. Copy the base-64 encoded, encrypted token from step 1’s CLI output (CiphertextBlob attribute) and replace “your KMS encrypted key” in the script with this copied value.

Enter this command to install Zip:

 sudo apt-get install zip

Zip up your code:

zip -r Cloudwatch-metrics-to-loggly.zip index.js node_modules

The resulting zip (Cloudwatch-metrics-to-loggly.zip) is what you will upload to AWS in step 3 below.

3. Configure the Lambda Function

Sign in to your AWS account and open the IAM console – https://console.aws.amazon.com/iam/
In your IAM console:

– Create a new Role say, ‘CloudWatch-Full-Access-Role‘.

create role

– Select Role type as ‘Lambda‘ from the AWS Service Roles.

lambda

– Attach policy ‘CloudWatchFullAccess‘ and save.

attach policy

 

Go to AWS Lambda Console. Click the “Author from scratch” button under Lambda Functions.

 

Note: You may need a quick and easy way to get this zipped file on the local box.

Use the command below to copy from the remote instance to the local box:

scp -i mykey.pem somefile.txt root@my.ec2.id.amazonaws.com:/

Fill in the following details.

Type the name and Choose an existing role. Set the Role as created above: cloudwatch-full-access (you can name it differently as needed)

Select Node.js 6.10 in runtime Upload lambda function (zip file you made above in Step 2) Handler: index.handler

Set memory at 256MB Set Timer to 2 minutes.

Configure the Event Source to call Cloudwatch-metrics-to-loggly. Go to the AWS Lambda Console . Make sure the Cloudwatch-metrics-to-loggly lambda function is selected, then go to  ‘Triggers‘ tab and click on “Add trigger”. Then fill in the following details.

Event source type: CloudWatch Events – Schedule

Name: Name for event. Like (custom-schedule-five-minute – rate(5 minutes))

Description: Description for events.
Schedule expression: Select rate(5 minutes).
Enable event source: select Enable Now. Click on submit button.

4. Adding CloudWatchFullAccess to the list of IAM users and roles who can use the key:

You need to add CloudWatchFullAccess to the list of IAM users and roles who can use this key to encrypt and decrypt data from within the applications and when using AWS services integrated with KMS. Head to the IAM and add the CloudWatchFullAccess policy as shown below.

Important: If you miss this step you will not be able to successfully send Cloudwatch metrics to Loggly.

5. Verify Events

Search Loggly events with the tag CloudwatchMetrics over the past 20 minutes. It may take few minutes to index the events. If this doesn’t work, see the troubleshooting section below.

tag:CloudwatchMetrics
Cloudwatch Metrics

Advanced AWS CloudWatch Options

Troubleshooting

If you don’t see any data appear in the verification step, then check for these common problems.

  • Make sure you’ve included your own customer token.
  • You can use multiple ways to create roles in an AWS Service
  • Make sure you have configured the same roles as mentioned above.
  • Make sure you are using the latest version of AWS CLI.
  • Check the Cloudwatch Logs for the log group of your script to see if it’s running and if there are any errors.
  • Search or post your own Amazon CloudWatch Metrics logging questions in the community forum.
Thanks for the feedback! We'll use it to improve our support documentation.