Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
.github/workflows		.github/workflows
docs		docs
examples		examples
modules/serverless		modules/serverless
.editorconfig		.editorconfig
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.releaserc.json		.releaserc.json
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
kafka_connect.tf		kafka_connect.tf
main.tf		main.tf
outputs.tf		outputs.tf
variables.tf		variables.tf
versions.tf		versions.tf

Repository files navigation

AWS MSK Kafka Cluster Terraform module

Terraform module which creates AWS MSK (Managed Streaming for Kafka) resources.

Usage

Seeexamples directory for working examples to reference:

module"msk_kafka_cluster" {source="terraform-aws-modules/msk-kafka-cluster/aws"name=local.namekafka_version="3.5.1"number_of_broker_nodes=3enhanced_monitoring="PER_TOPIC_PER_PARTITION"broker_node_client_subnets=["subnet-12345678","subnet-024681012","subnet-87654321"]broker_node_storage_info={    ebs_storage_info= { volume_size=100 }  }broker_node_instance_type="kafka.t3.small"broker_node_security_groups=["sg-12345678"]encryption_in_transit_client_broker="TLS"encryption_in_transit_in_cluster=trueconfiguration_name="example-configuration"configuration_description="Example configuration"configuration_server_properties={"auto.create.topics.enable"=true"delete.topic.enable"=true  }jmx_exporter_enabled=truenode_exporter_enabled=truecloudwatch_logs_enabled=trues3_logs_enabled=trues3_logs_bucket="aws-msk-kafka-cluster-logs"s3_logs_prefix=local.namescaling_max_capacity=512scaling_target_value=80client_authentication={    sasl= { scram=true }  }create_scram_secret_association=truescram_secret_association_secret_arn_list=[aws_secretsmanager_secret.one.arn,aws_secretsmanager_secret.two.arn,  ]# AWS Glue Registryschema_registries={    team_a= {      name="team_a"      description="Schema registry for Team A"    }    team_b= {      name="team_b"      description="Schema registry for Team B"    }  }# AWS Glue Schemasschemas={    team_a_tweets= {      schema_registry_name="team_a"      schema_name="tweets"      description="Schema that contains all the tweets"      compatibility="FORWARD"      schema_definition="{\"type\":\"record\",\"name\":\"r1\",\"fields\": [ {\"name\":\"f1\",\"type\":\"int\"}, {\"name\":\"f2\",\"type\":\"string\"} ]}"      tags= { Team="Team A" }    }    team_b_records= {      schema_registry_name="team_b"      schema_name="records"      description="Schema that contains all the records"      compatibility="FORWARD"      team_b_records= {        schema_registry_name="team_b"        schema_name="records"        description="Schema that contains all the records"        compatibility="FORWARD"        schema_definition=jsonencode({          type="record"          name="r1"          fields= [            {              name="f1"              type="int"            },            {              name="f2"              type="string"            },            {              name="f3"              type="boolean"            }          ]        })        tags= { Team="Team B" }      }    }  }tags={    Terraform="true"    Environment="dev"  }}

Examples

Examples codified under theexamples are intended to give users references for how to use the module(s) as well as testing/validating changes to the source code of the module. If contributing to the project, please be sure to make any appropriate updates to the relevant examples to allow maintainers to test your changes and to keep the examples up to date for users. Thank you!

Requirements

Name	Version
terraform	>= 1.5.7
aws	>= 6.22.1
random	>= 3.6

Providers

Name	Version
aws	>= 6.22.1
random	>= 3.6

Modules

No modules.

Resources

Name	Type
aws_appautoscaling_policy.this	resource
aws_appautoscaling_target.this	resource
aws_cloudwatch_log_group.this	resource
aws_glue_registry.this	resource
aws_glue_schema.this	resource
aws_msk_cluster.this	resource
aws_msk_cluster_policy.this	resource
aws_msk_configuration.this	resource
aws_msk_scram_secret_association.this	resource
aws_msk_vpc_connection.this	resource
aws_mskconnect_custom_plugin.this	resource
aws_mskconnect_worker_configuration.this	resource
random_id.this	resource
aws_iam_policy_document.this	data source

Inputs

Name	Description	Type	Default	Required
broker_node_az_distribution	The distribution of broker nodes across availability zones (documentation). Currently the only valid value is`DEFAULT`	`string`	`null`	no
broker_node_client_subnets	A list of subnets to connect to in client VPC (documentation)	`list(string)`	`[]`	no
broker_node_connectivity_info	Information about the cluster access configuration	object({ vpc_connectivity = optional(object({ client_authentication = optional(object({ sasl = optional(object({ iam = optional(bool) scram = optional(bool) })) tls = optional(bool) })) })) public_access = optional(object({ type = optional(string) })) })	`null`	no
broker_node_instance_type	Specify the instance type to use for the kafka brokers. e.g. kafka.m5.large. (Pricing info)	`string`	`null`	no
broker_node_security_groups	A list of the security groups to associate with the elastic network interfaces to control who can communicate with the cluster	`list(string)`	`[]`	no
broker_node_storage_info	A block that contains information about storage volumes attached to MSK broker nodes	object({ ebs_storage_info = optional(object({ provisioned_throughput = optional(object({ enabled = optional(bool) volume_throughput = optional(number) })) volume_size = optional(number, 64) })) })	`null`	no
client_authentication	Configuration block for specifying a client authentication	object({ sasl = optional(object({ iam = optional(bool) scram = optional(bool) })) tls = optional(object({ certificate_authority_arns = optional(list(string)) })) unauthenticated = optional(bool) })	`null`	no
cloudwatch_log_group_class	Specifies the log class of the log group. Possible values are: STANDARD or INFREQUENT_ACCESS.	`string`	`null`	no
cloudwatch_log_group_kms_key_id	The ARN of the KMS Key to use when encrypting log data	`string`	`null`	no
cloudwatch_log_group_name	Name of the Cloudwatch Log Group to deliver logs to	`string`	`null`	no
cloudwatch_log_group_retention_in_days	Specifies the number of days you want to retain log events in the log group	`number`	`0`	no
cloudwatch_logs_enabled	Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs	`bool`	`false`	no
cluster_override_policy_documents	Override policy documents for cluster policy	`list(string)`	`null`	no
cluster_policy_statements	Map of policy statements for cluster policy	map(object({ sid = optional(string) actions = optional(list(string)) not_actions = optional(list(string)) effect = optional(string) resources = optional(list(string)) not_resources = optional(list(string)) principals = optional(list(object({ type = string identifiers = list(string) }))) not_principals = optional(list(object({ type = string identifiers = list(string) }))) condition = optional(list(object({ test = string values = list(string) variable = string }))) }))	`null`	no
cluster_source_policy_documents	Source policy documents for cluster policy	`list(string)`	`null`	no
configuration_arn	ARN of an externally created configuration to use	`string`	`null`	no
configuration_description	Description of the configuration	`string`	`null`	no
configuration_name	Name of the configuration	`string`	`null`	no
configuration_revision	Revision of the externally created configuration to use	`number`	`null`	no
configuration_server_properties	Contents of the server.properties file. Supported properties are documented in theMSK Developer Guide	`map(string)`	`{}`	no
connect_custom_plugins	Map of custom plugin configuration details (map of maps)	map(object({ name = string description = optional(string) content_type = string s3_bucket_arn = string s3_file_key = string s3_object_version = optional(string) timeouts = optional(object({ create = optional(string) delete = optional(string) })) }))	`{}`	no
connect_worker_config_description	A summary description of the worker configuration	`string`	`null`	no
connect_worker_config_name	The name of the worker configuration	`string`	`null`	no
connect_worker_config_properties_file_content	Contents of connect-distributed.properties file. The value can be either base64 encoded or in raw format	`string`	`null`	no
create	Determines whether cluster resources will be created	`bool`	`true`	no
create_cloudwatch_log_group	Determines whether to create a CloudWatch log group	`bool`	`true`	no
create_cluster_policy	Determines whether to create an MSK cluster policy	`bool`	`false`	no
create_configuration	Determines whether to create a configuration	`bool`	`true`	no
create_connect_worker_configuration	Determines whether to create connect worker configuration	`bool`	`false`	no
create_schema_registry	Determines whether to create a Glue schema registry for managing Avro schemas for the cluster	`bool`	`true`	no
create_scram_secret_association	Determines whether to create SASL/SCRAM secret association	`bool`	`false`	no
enable_storage_autoscaling	Determines whether autoscaling is enabled for storage	`bool`	`true`	no
encryption_at_rest_kms_key_arn	You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest	`string`	`null`	no
encryption_in_transit_client_broker	Encryption setting for data in transit between clients and brokers. Valid values:`TLS`,`TLS_PLAINTEXT`, and`PLAINTEXT`. Default value is`TLS`	`string`	`null`	no
encryption_in_transit_in_cluster	Whether data communication among broker nodes is encrypted. Default value:`true`	`bool`	`null`	no
enhanced_monitoring	Specify the desired enhanced MSK CloudWatch monitoring level. SeeMonitoring Amazon MSK with Amazon CloudWatch	`string`	`null`	no
firehose_delivery_stream	Name of the Kinesis Data Firehose delivery stream to deliver logs to	`string`	`null`	no
firehose_logs_enabled	Indicates whether you want to enable or disable streaming broker logs to Kinesis Data Firehose	`bool`	`false`	no
jmx_exporter_enabled	Indicates whether you want to enable or disable the JMX Exporter	`bool`	`false`	no
kafka_version	Specify the desired Kafka software version	`string`	`null`	no
name	Name of the MSK cluster	`string`	`"msk"`	no
node_exporter_enabled	Indicates whether you want to enable or disable the Node Exporter	`bool`	`false`	no
number_of_broker_nodes	The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets	`number`	`null`	no
rebalancing	Configuration block for intelligent rebalancing	object({ status = string })	`null`	no
region	Region where this resource will be managed. Defaults to the Region set in the provider configuration	`string`	`null`	no
s3_logs_bucket	Name of the S3 bucket to deliver logs to	`string`	`null`	no
s3_logs_enabled	Indicates whether you want to enable or disable streaming broker logs to S3	`bool`	`false`	no
s3_logs_prefix	Prefix to append to the folder name	`string`	`null`	no
scaling_max_capacity	Max storage capacity for Kafka broker autoscaling	`number`	`250`	no
scaling_role_arn	The ARN of the IAM role that allows Application AutoScaling to modify your scalable target on your behalf. This defaults to an IAM Service-Linked Role	`string`	`null`	no
scaling_target_value	The Kafka broker storage utilization at which scaling is initiated	`number`	`70`	no
schema_registries	A map of schema registries to be created	map(object({ name = string description = optional(string) tags = optional(map(string), {}) }))	`{}`	no
schemas	A map schemas to be created within the schema registry	map(object({ schema_name = string schema_registry_name = string description = optional(string) data_format = optional(string, "AVRO") compatibility = string schema_definition = string tags = optional(map(string), {}) }))	`{}`	no
scram_secret_association_secret_arn_list	List of AWS Secrets Manager secret ARNs to associate with SCRAM	`list(string)`	`[]`	no
storage_mode	Controls storage mode for supported storage tiers. Valid values are:`LOCAL` or`TIERED`	`string`	`null`	no
tags	A map of tags to assign to the resources created	`map(string)`	`{}`	no
timeouts	Create, update, and delete timeout configurations for the cluster	object({ create = optional(string) update = optional(string) delete = optional(string) })	`null`	no
vpc_connections	Map of VPC Connections to create	map(object({ authentication = string client_subnets = list(string) security_groups = list(string) vpc_id = string tags = optional(map(string), {}) }))	`{}`	no

Outputs

Name	Description
appautoscaling_policy_arn	The ARN assigned by AWS to the scaling policy
appautoscaling_policy_name	The scaling policy's name
appautoscaling_policy_policy_type	The scaling policy's type
arn	Amazon Resource Name (ARN) of the MSK cluster
bootstrap_brokers	Comma separated list of one or more hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka cluster
bootstrap_brokers_plaintext	Comma separated list of one or more hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka cluster. Contains a value if`encryption_in_transit_client_broker` is set to`PLAINTEXT` or`TLS_PLAINTEXT`
bootstrap_brokers_public	Comma separated list of one or more hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka cluster
bootstrap_brokers_public_sasl_iam	One or more DNS names (or IP addresses) and SASL IAM port pairs
bootstrap_brokers_public_sasl_scram	One or more DNS names (or IP addresses) and SASL SCRAM port pairs
bootstrap_brokers_public_tls	One or more DNS names (or IP addresses) and TLS port pairs
bootstrap_brokers_sasl_iam	One or more DNS names (or IP addresses) and SASL IAM port pairs. This attribute will have a value if`encryption_in_transit_client_broker` is set to`TLS_PLAINTEXT` or`TLS` and`client_authentication_sasl_iam` is set to`true`
bootstrap_brokers_sasl_scram	One or more DNS names (or IP addresses) and SASL SCRAM port pairs. This attribute will have a value if`encryption_in_transit_client_broker` is set to`TLS_PLAINTEXT` or`TLS` and`client_authentication_sasl_scram` is set to`true`
bootstrap_brokers_tls	One or more DNS names (or IP addresses) and TLS port pairs. This attribute will have a value if`encryption_in_transit_client_broker` is set to`TLS_PLAINTEXT` or`TLS`
bootstrap_brokers_vpc_connectivity	Comma separated list of one or more hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka cluster
bootstrap_brokers_vpc_connectivity_sasl_iam	One or more DNS names (or IP addresses) and SASL IAM port pairs for VPC connectivity
bootstrap_brokers_vpc_connectivity_sasl_scram	One or more DNS names (or IP addresses) and SASL SCRAM port pairs for VPC connectivity
bootstrap_brokers_vpc_connectivity_tls	One or more DNS names (or IP addresses) and TLS port pairs for VPC connectivity
cluster_name	Name of the MSK cluster
cluster_uuid	UUID of the MSK cluster, for use in IAM policies
configuration_arn	Amazon Resource Name (ARN) of the configuration
configuration_latest_revision	Latest revision of the configuration
connect_custom_plugins	A map of output attributes for the connect custom plugins created
connect_worker_configuration_arn	The Amazon Resource Name (ARN) of the worker configuration
connect_worker_configuration_latest_revision	An ID of the latest successfully created revision of the worker configuration
current_version	Current version of the MSK Cluster used for updates, e.g.`K13V1IB3VIYZZH`
log_group_arn	The Amazon Resource Name (ARN) specifying the log group
schema_registries	A map of output attributes for the schema registries created
schemas	A map of output attributes for the schemas created
scram_secret_association_id	Amazon Resource Name (ARN) of the MSK cluster
vpc_connections	A map of output attributes for the VPC connections created
zookeeper_connect_string	A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster. The returned values are sorted alphabetically
zookeeper_connect_string_tls	A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster via TLS. The returned values are sorted alphabetically