diff --git a/README.md b/README.md index 8edc4b8..d3f6bb2 100644 --- a/README.md +++ b/README.md @@ -4,60 +4,58 @@ Sends logs from Cloudwatch logs to Loggly using Lamda function ## More information about AWS Lambda and Loggly * http://aws.amazon.com/lambda/ * https://www.loggly.com/ - -## Get the code and prepare it for the uploading to AWS -* Clone the git repo -```bash -git clone https://github.com/psquickitjayant/cloudwatch2loggly.git -cd cloudwatch2loggly -``` -* Install required npm packages. -``` -npm install -``` -* zip up your code -```bash -zip -r cloudwatch2loggly.zip index.js node_modules -``` - -The resulting zip (cloudwatch2loggly.zip) is what you will upload to AWS. - -## Setting up AWS -For all of the AWS setup, I used the AWS console following [this -example](http://docs.aws.amazon.com/lambda/latest/dg/getting-started-amazons3-events.html). Below, you will find a high-level -description of how to do this. I also found [this blog post](http://alestic.com/2014/11/aws-lambda-cli) on how to set things up -using the command line tools. - -### Create and upload the cloudwatch2loggly function in the AWS Console +## Getting started with AWS Lambda +Getting started documentation for AWS lambda can be found in [this +article](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html). There is also [this blog post](http://alestic.com/2014/11/aws-lambda-cli) on how to set things up using the command line tools. + +## Creating lambda function in AWS +1. Build lambda function code + * Clone the Git repository: + `git clone https://github.com/loggly/cloudwatch2loggly.git` + * Go to cloned folder: + `cd cloudwatch2loggly` + * Install dependencies: + `npm install` + * Create a .zip file for upload to AWS console later: + `zip -r cloudwatch2loggly.zip index.js node_modules` 1. Create Role - 1. Sign in to your AWS account and open IAM console https://console.aws.amazon.com/iam/ - 2. In your IAM console create a new Role say, 'cloudwatch-full-access' - 3. Select Role Type as 'AWS Lambda' - 4. Apply policy 'CloudWatchFullAccess' and save. -2. Create KMS Key - 1. Create a KMS key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html - 2. Encrypt the Loggly Customer Token using the AWS CLI - **aws kms encrypt --key-id alias/<your KMS key arn> --plaintext "<your loggly customer token>"** - 3. Copy the base-64 encoded, encrypted token from step 2's CLI output (CiphertextBlob attribute) and replace it with the "your KMS encypted key" in the script at line no 22 -3. Create lambda function - 1. https://console.aws.amazon.com/lambda/home - 2. Click "Create a Lambda function" button. *(Choose "Upload a .ZIP file")* - * **Name:** *cloudwatch2loggly* - * Upload lambda function (zip file you made above.) - * **Handler*:** *index.handler* - * Set Role : *cloudwatch-full-access* - * Set Timeout to 2 minutes - 3. Go to your Lamda function and select the "Event sources" tab - * Click on **Add Event Source** - * Event Source Type : *CloudWatch Logs* - * Log Group : Select your log group whose logs you want to send to Loggly. - * Filter Name: Provide your filter name. - * Filter Pattern: This is not a mandatory field. You can keep it empty. - * Enable Event Source : *Enable Now* - Now click on submit and wait for the events to occur in Loggly + * Sign in to your AWS account and open IAM console https://console.aws.amazon.com/iam/ + * In your IAM console create a new Role say, `cloudwatch-full-access` + * Select Role Type as **AWS Lambda** + * Apply policy **CloudWatchFullAccess** and save. +1. Create KMS Key + * Create a KMS key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html + * Encrypt the Loggly Customer Token using the AWS CLI + `aws kms encrypt --key-id alias/ --plaintext ""` + * Copy or keep `CiphertextBlob` attribute for furhter usage in the next step. +1. Create lambda function + * Go to https://console.aws.amazon.com/lambda/home + * Click **Create a Lambda function** button. + * Select **Author from scratch** option + * Set **Function name** for example to `cloudwatch2loggly` + * Set **Runtime** to `Node.js 12.x` + * Under **Permissions** click on **Choose or create an execution role** + * Select **Use an existing role** and select **cloudwatch-full-access** role created above in step 1 + * Click on **Create function** button + * Scroll to **Function code** section + * Select **Upload a .zip file** in **Code entry type** dropdown + * Upload lambda function (zip file `cloudwatch2loggly.zip` you made above) + * Go to **Environment variables** section + * Define new environment variable **kmsEncryptedCustomerToken** and set it to `CiphertextBlob` value from step 3 above (Create KMS Key) + * Scroll to **Basic settings** section + * Set **Memory (MB)** to **512 MB** + * Set Timeout to **2** minutes + * Scroll up to **Designer** section (expand if it's collapsed) + * Click on **Add trigger** button + * Select on **CloudWatch Logs** + * Select appropriate **Log group** + * Name the filter + * Make sure the checkbox **Enable trigger** is checked + * Click on **Add** + * Click on **Save** to save the whole lambda function. + * Wait for the events to occur in Loggly **NOTE**: Always use latest version of **AWSCLI**. Some features like KMS may not work on older versions of AWSCLI. To upgrade, use the command given below `pip install --upgrade awscli` - - diff --git a/index.js b/index.js index 03747f9..4c63a90 100644 --- a/index.js +++ b/index.js @@ -1,37 +1,40 @@ -/** - * To setup your encrypted Loggly Customer Token inside the script use the following steps - * 1. Create a KMS key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html - * 2. Encrypt the Loggly Customer Token using the AWS CLI - * aws kms encrypt --key-id alias/ --plaintext "" - * 3. Copy the base-64 encoded, encrypted token from step 2's CLI output (CiphertextBlob attribute) and - * paste it in place of the 'your KMS encypted key' below in line 27 - */ - -var AWS = require('aws-sdk'), - http = require('http'), - zlib = require('zlib'); +'use strict'; + +/* + * To encrypt your secrets use the following steps: + * + * 1. Create or use an existing KMS Key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html + * + * 2. Expand "Encryption configuration" and click the "Enable helpers for encryption in transit" checkbox + * + * 3. Paste into the kmsEncryptedCustomerToken environment variable and click "Encrypt" + * + * 4. Give your function's role permission for the `kms:Decrypt` action using the provided policy template +*/ + +const AWS = require('aws-sdk'); +const http = require('http'); +const zlib = require('zlib'); + // loggly url, token and tag configuration -// user need to edit while uploading code via blueprint -var logglyConfiguration = { - hostName: 'logs-01.loggly.com', - tags: 'CloudWatch2Loggly' +// user needs to edit environment variables when creating function via blueprint +// logglyHostName, e.g. logs-01.loggly.com +// logglyTags, e.g. CloudWatch2Loggly +const logglyConfiguration = { + hostName: process.env.logglyHostName || 'logs-01.loggly.com', + tags: process.env.logglyTags || 'CloudWatch2Loggly', }; -var cloudWatchLogs = new AWS.CloudWatchLogs({ - apiVersion: '2014-03-28' -}); - -// use KMS to decrypt customer token -var decryptParams = { - CiphertextBlob: new Buffer('your KMS encypted key', 'base64') +// use KMS to decrypt customer token in kmsEncryptedCustomerToken environment variable +const decryptParams = { + CiphertextBlob: Buffer.from(process.env.kmsEncryptedCustomerToken, 'base64'), + EncryptionContext: { LambdaFunctionName: process.env.AWS_LAMBDA_FUNCTION_NAME }, }; -var kms = new AWS.KMS({ - apiVersion: '2014-11-01' -}); +const kms = new AWS.KMS({ apiVersion: '2014-11-01' }); -kms.decrypt(decryptParams, function (error, data) { +kms.decrypt(decryptParams, (error, data) => { if (error) { logglyConfiguration.tokenInitError = error; console.log(error); @@ -41,30 +44,17 @@ kms.decrypt(decryptParams, function (error, data) { }); // entry point -exports.handler = function (event, context) { - var payload = new Buffer(event.awslogs.data, 'base64'); - - zlib.gunzip(payload, function (error, result) { - if (error) { - context.fail(error); - } else { - var result_parsed = JSON.parse(result.toString('ascii')); - var parsedEvents = result_parsed.logEvents.map(function(logEvent) { - return parseEvent(logEvent, result_parsed.logGroup, result_parsed.logStream); - }); - - postEventsToLoggly(parsedEvents); - } - }); +exports.handler = (event, context, callback) => { + const payload = Buffer.from(event.awslogs.data, 'base64'); // converts the event to a valid JSON object with the sufficient infomation required function parseEvent(logEvent, logGroupName, logStreamName) { return { // remove '\n' character at the end of the event message: logEvent.message.trim(), - logGroupName: logGroupName, - logStreamName: logStreamName, - timestamp: new Date(logEvent.timestamp).toISOString() + logGroupName, + logStreamName, + timestamp: new Date(logEvent.timestamp).toISOString(), }; } @@ -74,59 +64,70 @@ exports.handler = function (event, context) { if (!logglyConfiguration.customerToken) { if (logglyConfiguration.tokenInitError) { console.log('error in decrypt the token. Not retrying.'); - return context.fail(logglyConfiguration.tokenInitError); + return callback(logglyConfiguration.tokenInitError); } console.log('Cannot flush logs since authentication token has not been initialized yet. Trying again in 100 ms.'); - setTimeout(function () { postEventsToLoggly(parsedEvents) }, 100); + setTimeout(() => postEventsToLoggly(parsedEvents), 100); return; } // get all the events, stringify them and join them // with the new line character which can be sent to Loggly // via bulk endpoint - var finalEvent = parsedEvents.map(JSON.stringify).join('\n'); + const finalEvent = parsedEvents.map(JSON.stringify).join('\n'); // creating logglyURL at runtime, so that user can change the tag or customer token in the go // by modifying the current script // create request options to send logs try { - var options = { + const options = { hostname: logglyConfiguration.hostName, - path: '/bulk/' + logglyConfiguration.customerToken + '/tag/' + encodeURIComponent(logglyConfiguration.tags), + path: `/bulk/${logglyConfiguration.customerToken}/tag/${encodeURIComponent(logglyConfiguration.tags)}`, method: 'POST', headers: { 'Content-Type': 'application/json', - 'Content-Length': finalEvent.length - } + 'Content-Length': finalEvent.length, + }, }; - var req = http.request(options, function (res) { - res.on('data', function (result) { - result = JSON.parse(result.toString()); + const req = http.request(options, (res) => { + res.on('data', (data) => { + const result = JSON.parse(data.toString()); if (result.response === 'ok') { - context.succeed('all events are sent to Loggly'); + callback(null, 'all events are sent to Loggly'); } else { console.log(result.response); } }); - res.on('end', function () { + res.on('end', () => { console.log('No more data in response.'); - context.done(); + callback(); }); }); - req.on('error', function (e) { - console.log('problem with request: ' + e.toString()); - context.fail(e); + req.on('error', (err) => { + console.log('problem with request:', err.toString()); + callback(err); }); // write data to request body req.write(finalEvent); req.end(); - } catch (ex) { console.log(ex.message); - context.fail(ex.message); + callback(ex.message); } } + + zlib.gunzip(payload, (error, result) => { + if (error) { + callback(error); + } else { + const resultParsed = JSON.parse(result.toString('ascii')); + const parsedEvents = resultParsed.logEvents.map((logEvent) => + parseEvent(logEvent, resultParsed.logGroup, resultParsed.logStream)); + + postEventsToLoggly(parsedEvents); + } + }); }; diff --git a/package.json b/package.json index 8de4127..5d07090 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "cloudwatch2loggly", - "version": "1.0.0", + "version": "1.1.0", "description": "A NodeJS library to send Cloudwatch logs to Loggly", "main": "index.js", "scripts": { @@ -8,22 +8,20 @@ }, "repository": { "type": "git", - "url": "https://github.com/psquickitjayant/cloudwatch2loggly.git" + "url": "https://github.com/loggly/cloudwatch2loggly.git" }, "keywords": [ "Loggly", "Cloudwatch" ], - "author": "psquickitjayant", + "author": "loggly", "license": "MIT", "bugs": { - "url": "https://github.com/psquickitjayant/cloudwatch2loggly/issues" + "url": "https://github.com/loggly/cloudwatch2loggly/issues" }, - "homepage": "https://github.com/psquickitjayant/cloudwatch2loggly", + "homepage": "https://github.com/loggly/cloudwatch2loggly", "dependencies": { - "aws-sdk": "^2.2.25", - "q": "^1.4.1", - "request": "^2.67.0", + "aws-sdk": "^2.754.0", "zlib": "^1.0.5" } }