Repository files navigation

Terraform module to provision a scheduled Lambda function which willdelete old Elasticsearch indexes usingSigV4Auth authentication. Thelambda function can optionally send output to an SNS topic if thetopic ARN is given. This module was largely inspired byaws-lambda-es-cleanup

Tip

👽 Use Atmos with Terraform

Cloud Posse usesatmos to easily orchestrate multiple environments using Terraform.
Works withGithub Actions,Atlantis, orSpacelift.

Watch demo of using Atmos with Terraform

Example of runningatmos to manage infrastructure from ourQuick Start tutorial.

For a complete example, seeexamples/complete.

For automated tests of the complete example usingbats andTerratest (which tests and deploys the example on AWS), seetest.

module"elasticsearch_cleanup" {source="https://github.com/cloudposse/terraform-aws-lambda-elasticsearch-cleanup.git?ref=master"es_endpoint=module.elasticsearch.domain_endpointes_domain_arn=module.elasticsearch.domain_arnes_security_group_id=module.elasticsearch.security_group_idvpc_id=module.vpc.vpc_idnamespace="eg"stage="dev"schedule="cron(0 3 * * ? *)"}

Indexes are expected to be in the formatname-date wheredate is in the format specified byvar.index_format.By default, all indexes except for the ones added by Kibana will be deleted based on the date part of the fullindex name. The actual creation date of the index is not used.

Index matching is done with unanchored regular expresssion, so "bar" matches index "foobarbaz".

• If the full index name, including the date part, matchesskip_index_re, then the index will be skipped (never deleted).Kibana indexes are skipped by the defaultskip_index_re of^\.kibana* so if you specify a value forskip_index_reyou must include the Kibana exception in your regex if you want it excepted. (Since Kibana indexes do not have adate part, this module should not delete them, but will complain about them having malformed dates if they are not excluded.)
• If the index name without the trailing-date part matchesindex_re, then it will be cleaned up according to the date part.

Keep in mind that, fundamentally, this module expects indexes to be in the format ofname-date so it will not workproperly if the regexes end up selecting an index that does not end with-date. To avoid edge cases, it is wise notto include dashes in your index name or date format.

Prior to version 0.10.0, this moudle had inputsindex, which was a comma-separated list of index names or thespecial name "all" to indicate all but Kibana indexes, andindex_regex, which was a regular expression for parsingindex name and date parts. There was no mechanism for specifying a list of indexes to exclude.Starting with version 0.10.0 this module drops those inputs and instead takesindex_re andskip_index_re,both of which are regular expressions. (You probably want to anchor your regexes to the beginning of the index nameby starting with^).

This module creates a scheduled Lambda function which will delete oldElasticsearch indexes using SigV4Auth authentication. The lambdafunction can optionally send output to an SNS topic if the topic ARNis given

{
  "additional_tag_map": {},
  "attributes": [],
  "delimiter": null,
  "descriptor_formats": {},
  "enabled": true,
  "environment": null,
  "id_length_limit": null,
  "label_key_case": null,
  "label_order": [],
  "label_value_case": null,
  "labels_as_tags": [
    "unset"
  ],
  "name": null,
  "namespace": null,
  "regex_replace_chars": null,
  "stage": null,
  "tags": {},
  "tenant": null
}

[
  "default"
]

Check out these related projects.

• terraform-aws-vpc - Terraform Module that defines a VPC with public/private subnets across multiple AZs with Internet Gateways
• terraform-aws-dynamic-subnets - Terraform module for dynamic subnets provisioning.
• terraform-aws-elasticsearch - Terraform module for AWS Elasticsearch provisioning.

This project is under active development, and we encourage contributions from our community.

Many thanks to our outstanding contributors:

For 🐛 bug reports & feature requests, please use theissue tracker.

In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.

1. Review ourCode of Conduct andContributor Guidelines.
2. Fork the repo on GitHub
3. Clone the project to your own machine
4. Commit changes to your own branch
5. Push your work back up to your fork
6. Submit aPull Request so that we can review your changes

NOTE: Be sure to merge the latest changes from "upstream" before making a pull request!

We useAtmos to streamline how Terraform tests are run. It centralizes configuration and wraps common test workflows with easy-to-use commands.

All tests are located in thetest/ folder.

Under the hood, tests are powered by Terratest together with our internalTest Helpers library, providing robust infrastructure validation.

Setup dependencies:

• Install Atmos (installation guide)
• Install Go1.24+ or newer
• Install Terraform or OpenTofu

To run tests:

• Run all tests:

  atmostest run

• Clean up test artifacts:

  atmostest clean

• Explore additional test options:

  atmostest --help

The configuration for test commands is centrally managed. To review what's being imported, see theatmos.yaml file.

Learn more about ourautomated testing in our documentation or implementingcustom commands with atmos.

Join ourOpen Source Community on Slack. It'sFREE for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totallysweet infrastructure.

Sign up forour newsletter and join 3,000+ DevOps engineers, CTOs, and founders who get insider access to the latest DevOps trends, so you can always stay in the know.Dropped straight into your Inbox every week — and usually a 5-minute read.

Join us every Wednesday via Zoom for your weekly dose of insider DevOps trends, AWS news and Terraform insights, all sourced from our SweetOps community, plus alive Q&A that you can’t find anywhere else.It'sFREE for everyone!

Licensed to the Apache Software Foundation (ASF) under oneor more contributor license agreements.  See the NOTICE filedistributed with this work for additional informationregarding copyright ownership.  The ASF licenses this fileto you under the Apache License, Version 2.0 (the"License"); you may not use this file except in compliancewith the License.  You may obtain a copy of the License at  https://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing,software distributed under the License is distributed on an"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANYKIND, either express or implied.  See the License for thespecific language governing permissions and limitationsunder the License.

All other trademarks referenced herein are the property of their respective owners.

Copyright © 2017-2025Cloud Posse, LLC