Repository files navigation

This is the implementation for the solution architecture introduced in the blog post aboutmonitoring the Java Virtual Machine garbage collection on AWS Lambda.

When you want to optimize your Java application onAWS Lambda for performance and cost the general steps are: Build, measure, then optimize! To accomplish this, you need a solid monitoring mechanism.Amazon CloudWatch andAWS X-Ray are well suited for this task since they already provide lots of data about your AWS Lambda function. This includes overall memory consumption, initialization time, and duration of your invocations. To examine the Java Virtual Machine (JVM) memory you require garbage collection logs from your functions. Instances of an AWS Lambda function have a short lifecycle compared to a long-running Java application server. It can be challenging to process the logs from tens or hundreds of these instances.

With this solution you emit and collect data to monitor the JVM garbage collector activity. Having this data, you can visualize out-of-memory situations of your applications in a Kibana dashboard like in the following screenshot. You gain actionable insights into your application’s memory consumption on AWS Lambda for troubleshooting and optimization.

To start, launch the solution architecture described above as a prepackaged application from the AWS Serverless Application Repository. It contains all resources ready to visualize the garbage collection logs for your Java 11 AWS Lambda functions in a Kbana dashboard. The search cluster consists of a singlet3.small.elasticsearch instance with 10GB of EBS storage. It is protected with Amazon Cognito User Pools so you only need to add your user(s). The application template prefixes the search domain and the Amazon Cognito Hosted UI with a string that you can define with theapplicationPrefix template parameter.

You deploy the template through the AWS Cloud Development Kit (CDK). You must install and configure theAWS CLI and theCDK.

The CDK template is written in TypeScript. TypeScript sources must be compiled to JavaScript initially and after each modification. Open a new terminal and keep this terminal open in the background if you like to change the source files. Change the directory to the one wherecdk.json is and execute:

npm installnpm run watch

Read theCDK developer guide for more information.

Create or update the application withcdk deploy.

cdk deploy --parameters applicationPrefix=<PREFIX>

1. As soon as the application is deployed completely the outputs of theAWS CloudFormation stack provides the links for the next steps. You will find two URLs in the AWS CloudFormation console calledcreateUserUrl andkibanaUrl.

2. Use thecreateUserUrl link from the outputs, or navigate to theAmazon Cognito user pool in the console to create a new user in the pool.

   • Enter an email address as username and email. Enter a temporary password of your choice with at least 8 characters.
   • Leave the phone number empty and uncheck the checkbox to mark the phone number as verified.
   • If necessary, you can check the checkboxes to send an invitation to the new user or to make the user verify the email address.
   • ChooseCreate user.

3. Access the Kibana dashboard with thekibanaUrl link from the AWS CloudFormation stack outputs, or navigate to the Kibana link displayed in the Amazon Elasticsearch Service console.

   • In Kibana, choose the Dashboard icon in the left menu bar
   • Open theLambda GC Activity dashboard.

4. You can test that new events appear by using the Kibana Developer Console:

   POST gc-logs-2020.09.03/_doc{"@timestamp":"2020-09-03T15:12:34.567+0000","@gc_type":"Pause Young","@gc_cause":"Allocation Failure","@heap_before_gc":"2","@heap_after_gc":"1","@heap_size_gc":"9","@gc_duration":"5.432","@owner":"123456789012","@log_group":"/aws/lambda/myfunction","@log_stream":"2020/09/03/[$LATEST]123456"}

5. When you go to theLambda GC Activity dashboard you can see the new event. You must select the right timeframe with theShow dates link.

   

The dashboard consists of six tiles:

• In theFilters you can optionally select the log group and filter for a specific AWS Lambda function execution context by the name of its log stream.
• In theGC Activity Count by Execution Context you see a heatmap of all filtered execution contexts by garbage collection activity count.
• TheGC Activity Metrics display a graph for the metrics for all filtered execution contexts.
• TheGC Activity Count shows the amount of garbage collection activities that are currently displayed.
• TheGC Duration show the sum of the duration of all displayed garbage collection activities.
• TheGC Activity Raw Data at the bottom displays the raw items as ingested into the search cluster for a further drill down.

1. The application that you want to monitor needs to log garbage collection activities. Currently the solution supports logs fromJava 11. Add the following environment variable to your AWS Lambda function to activate the logging.

   JAVA_TOOL_OPTIONS=-Xlog:gc:stderr:time,tags

   Theenvironment variables must reflect this parameter like the following screenshot:

   

2. Go to thestreamLogs function in the AWS Lambda console that has been created by the stack, and subscribe it to the log group of the function you want to monitor.

   

3. SelectAdd Trigger.

4. SelectCloudWatch Logs asTrigger Configuration.

5. Input aFilter name of your choice.

6. Input"[gc" (including quotes) as theFilter pattern to match all garbage collection log entries.

7. Select theLog Group of the function you want to monitor. The following screenshot subscribes to the logs of the application’s functionresize-lambda-ResizeFn-[...].

   

8. SelectAdd.

9. Execute the AWS Lambda function you want to monitor.

10. Refresh the dashboard in Amazon Elasticsearch Service and see the datapoint added manually before appearing in the graph.

If you do not need the garbage collection monitoring anymore, delete the subscription filter from the log group of your AWS Lambda function(s). Also, delete the stack of the solution above in the AWS CloudFormation console to clean up resources.

1. Sign in to the AWS CloudFormation console and choose your stack.
2. Choose thestreamLogs AWS Lambda function so the function is displayed in the AWS Lambda console.
3. Delete all triggers in theDesigner panel.
4. Go back to the AWS CloudFormation console and your stack.
5. ChooseDelete to delete all resources, including the search cluster and the Amazon Cognito user pool.

The Launch Stack button above opens the AWS Serverless Application Repository in the US East 1 (Northern Virginia) region. You may switch to other regions from there before deployment.

Standard AWS charges apply to the resources you deploy with this template.

Cost for the processing and transformation of your function's Amazon CloudWatch Logs incurs when your function is called. This cost depends on your application and how often GC activities are triggered.

Amazon Elasticsearch Service provides customers in theAWS Free Tier free usage of up to 750 hours per month of the configuration in this template, i.e. a single-AZt3.small.elasticsearch instance and 10GB of EBS storage for up to one year from the date the account was created. If you exceed the free tier limits, you will be charged the Amazon Elasticsearch Service rates for the additional resources you use. Read anestimate of the monthly cost of the search cluster.

The Amazon Cognito User Pool feature has a free tier of 50,000 monthly active users for users who sign in directly to Cognito User Pools. The free tier does not automatically expire at the end of your 12 month AWS Free Tier term, and it is available to both existing and new AWS customers indefinitely.

See offer terms ofAmazon Cognito,Amazon Elasticsearch Service,AWS Lambda, andAmazon CloudWatch for more details.

See theContributing Guidelines for details.

SeeCONTRIBUTING for more information.

This library is licensed under the MIT-0 License. See theLICENSE file.