Send Amazon ELB Classic Logs to Loggly
You can push your Amazon Elastic Load Balancer (ELB) Classic logs to Loggly using an AWS Lambda Script. It converts the ELB logs written to S3 into JSON format and then sends them to Loggly. ELB (Application) Logs are not supported at this time.
Alternatively, you may use our S3 ingestion service that will directly ingest ELB Classic logs into Loggly without requiring a Lambda function. Our app pack for ELB contains popular dashboards and saved searches. It currently only supports ingesting logs using our S3 ingestion service and not this Lambda script.
1. Get the Lambda Code
Clone the git repo
git clone https://github.com/cboscolo/elb2loggly.git cd elb2loggly
Edit elb2loggly.js with proper Loggly customer token and optional log tags. (You can set these as tags on the S3 Bucket that contains the logs.) Install require npm packages.
Zip up your code
zip -r elb2loggly.zip elb2loggly.js node_modules
The resulting zip (elb2loggly.zip) is what you will upload to AWS in step 2 below.
2. Configure the Lambda Function
Go to AWS Lambda Console Console. Click “Create a Lambda function” button. (Choose “Upload a .ZIP file”). Fill the following details.
Name: elb2loggly Upload lambda function (zip file you made above in Step 1) Handler*: elb2loggly.handler Role*: In the drop down click "S3 execution role". (This will open a new window to create the role, click Allow) Set memory at 128MB Set Timer to 10 seconds.
Configure Event Source to call elb2loggly when logs added to S3 bucket. Go to AWS Lambda Console . Make sure the elb2loggly lambda function is selected, then click ‘Actions->Add event source‘. Then fill the following details.
Event source type: S3 Bucket: Choose the bucket that contains your ELB logs. Event type: ObjectCreated (All)
3. Configure ELB Logging
Goto the EC2 Management Console under ‘Load Balancers’. Choose your ELB and scroll down to Access Logs and click edit then set interval to 5 minutes and S3 Location to the bucket where you want to put your logs.
4. Tag Your Logs in S3
The Lamba script will look for your customer token as an S3 tag, which it will use to send data to your account. It will also add tags for Loggly, which will make the logs easier to find in search. Using the S3 Management Console click the bucket that contains your ELB logs. Under Properties -> Tags, add the following tag:
Key: loggly-customer-token Value: TOKEN Key: loggly-tag Value: aws-elb
- TOKEN: your customer token from the source setup page
5. Verify Events
Search Loggly events with the tag as aws-elb over the past 20 minutes. It may take few minutes to index the events. If if doesn’t work, see the troubleshooting section below.
If you don’t see any data show up in the verification step, then check for these common problems.
- Wait a few minutes in case indexing needs to catch up
- Make sure you’ve included your own customer token
- Make sure you have configured same roles as mentioned above.
- Go to your Lamda function in AWS Console and click on View logs in Cloudwatch in the Monitoring tab to view logs.
- If you still do not see your log, search for the “error” field with tag as aws-elb in your past 20 minutes logs.
- Search or post your own Amazon ELB logging questions in the community forum.