Refactor monolithic Terraform configuration

18min
|
Terraform
Interactive

Some Terraform projects start as amonolith, a Terraform project managed by a single main configuration file in a single directory, with a single state file. Small projects may be convenient to maintain this way. However, as your infrastructure grows, restructuring your monolith into logical units will make your Terraform configurations less confusing and safer to manage.

These tutorials are for Terraform users who need to restructure Terraform configurations as they grow. In this tutorial, you will provision two instances of a web application hosted in an S3 bucket that represent production and development environments. The configuration you use to deploy the application will start in as a monolith. You will modify it to step through the common phases of evolution for a Terraform project, until each environment has its own independent configuration and state.

Prerequisites

Although the concepts in this tutorial apply to any module creation workflow,this tutorial uses Amazon Web Services (AWS) modules.

To follow this tutorial you will need:

An AWS accountConfigure one of the authentication methods described in ourAWS ProviderDocumentation.The examples in this tutorial assume that you are using theShared Credentialsfilemethod with the default AWS credentials file and default profile.
TheAWS CLI
TheTerraform CLI

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Apply a monolith configuration

In your terminal, clone the examplerepository. It contains the configuration used in this tutorial.

$ git clone https://github.com/hashicorp-education/learn-terraform-code-organization

Tip

Throughout this tutorial, you will have the option to check out branches that correspond to the version of Terraform configuration in that section. You can use this as a failsafe if your deployment is not working correctly, or to run the tutorial without making changes manually.

Navigate to the directory.

$ cd learn-terraform-code-organization

Your root directory contains four files and an "assets" folder. The root directory files compose the configuration as well as the inputs and outputs of your deployment.

main.tf - configures the resources that make up your infrastructure.
variables.tf- declares input variables for yourdev andprod environment prefixes, and the AWS region to deploy to.
terraform.tfvars.example- defines your region and environment prefixes.
outputs.tf- specifies the website endpoints for yourdev andprod buckets.
assets- houses your webapp HTML file.

In your text editor, open themain.tf file. The file consists of a few different resources:

Therandom_pet resource creates a string to be used as part of the unique name of your S3 bucket.
Twoaws_s3_bucket resources designatedprod anddev, which each create an S3 bucket. Notice that thebucket argument defines the S3 bucket name by interpolating the environment prefix and therandom_pet resource name.
Twoaws_s3_bucket_acl resources designatedprod anddev, which set apublic-read ACL for your buckets.
Twoaws_s3_bucket_website_configuration resources designatedprod anddev, which configure your buckets to host websites.
Twoaws_s3_bucket_policy resources designatedprod anddev, which allow anyone to read the objects in the corresponding bucket.
Twoaws_s3_object resources designatedprod anddev, which load the file in the localassets directory using thebuilt infile()function and upload it to your S3 buckets.

Terraform requires unique identifiers - in this caseprod ordev for eachs3 resource - to create separate resources of the same type.

Open theterraform.tfvars.example file in your repository and edit it with your own variable definitions. Change theregion to your nearest location in your text editor.

terraform.tfvars.example

region= "us-east-1"prod_prefix= "prod"dev_prefix= "dev"

Save your changes in your editor and rename the file toterraform.tfvars. Terraform automatically loads variable values from any files that end in.tfvars.

$ mv terraform.tfvars.example terraform.tfvars

In your terminal, initialize your Terraform project.

$ terraform init

Then, apply the configuration.

$ terraform apply

Accept the apply plan by enteringyes in your terminal to create the 5 resources.

Navigate to the web address from the Terraform output to display the deployment in a browser. Your directory now contains a state file,terraform.tfstate.

Separate configuration

Defining multiple environments in the samemain.tf file may become hard to manage as you add more resources. The HashiCorp Configuration Language (HCL), which is the language used to write Terraform configurations, is meant to be human-readable and supports using multiple configuration files to help organize your infrastructure.

You will organize your current configuration by separating the configurations into two separate files — one root module for each environment.To split the configuration, first make a copy ofmain.tf and name itdev.tf.

$ cp main.tf dev.tf

Rename themain.tf file toprod.tf.

$ mv main.tf prod.tf

You now have two identical files. Opendev.tf and remove any references to the production environment by deleting the resource blocks with theprod ID. Repeat the process forprod.tf by removing any resource blocks with thedev ID.

Tip

To fast-forward to this file separated configuration, checkout the branch in your example repository by runninggit checkout file-separation.

Your directory structure will look similar to the one below.

.├── README.md├── assets│   └── index.html├── dev.tf├── outputs.tf├── prod.tf├── terraform.tfstate├── terraform.tfvars└── variables.tf

Although your resources are organized in environment-specific files, yourvariables.tf andterraform.tfvars files contain the variable declarations and definitions for both environments.

Terraform loads all configuration files within a directory and appends them together, so any resources or providers with the same name in the same directory will cause a validation error. If you were to run aterraform command now, yourrandom_pet resource andprovider block would cause errors since they are duplicated across the two files.

Edit theprod.tf file by commenting out theterraform block, theprovider block, and therandom_pet resource. You can comment out the configuration by adding a/* at the beginning of the commented out block and a*/ at the end, as shown below.

prod.tf

+/* terraform {   required_providers {     aws = {       source = "hashicorp/aws"       version = "~> 4.0.0"     }     random = {       source  = "hashicorp/random"       version = "~> 3.1.0"     }   } } provider "aws" {   region = var.region } resource "random_pet" "petname" {   length    = 3   separator = "-" }+*/

With yourprod.tf shared resources commented out, your production environment will still inherit the value of therandom_pet resource in yourdev.tf file.

Simulate a hidden dependency

You may want your development and production environments to share bucket names, but the current configuration is particularly dangerous because of the hidden resource dependency built into it. Imagine that you want to test a random pet name with four words in development. Indev.tf, update yourrandom_pet resource'slength attribute to4.

dev.tf

resource "random_pet" "random" {  length= 4  separator= "-"}

You might think you are only updating the development environment because you only changeddev.tf, but remember, this value is referenced by bothprod anddev resources.

$ terraform apply

Enteryes when prompted to apply the changes.

Note that the operation updated all five of your resources by destroying and recreating them. In this scenario, you encountered a hidden resource dependency because both bucket names rely on the same resource.

Carefully review Terraform execution plans before applying them. If an operator does not carefully review the plan output or if CI/CD pipelines automatically apply changes, you may accidentally apply breaking changes to your resources.

Destroy your resources before moving on. Respond to the confirmation prompt with ayes.

$ terraform destroy

Separate states

The previous operation destroyed both the development and production environment resources. When working with monolithic configuration, you can use theterraform apply command with the-target flag to scope the resources to operate on, but that approach can be risky and is not a sustainable way to manage distinct environments. For safer operations, you need to separate your development and production state.

State separation signals more mature usage of Terraform; with additional maturity comes additional complexity. There are two primary methods to separate state between environments: directories and workspaces.

To separate environments with potential configuration differences, use a directory structure. Use workspaces for environments that do not greatly deviate from one another, to avoid duplicating your configurations. Try both methods in the tabs below to understand which will serve your infrastructure best.

By creating separate directories for each environment, you can shrink the blast radius of your Terraform operations and ensure you will only modify intended infrastructure. Terraform stores your state files on disk in their corresponding configuration directories. Terraform operates only on the state and configuration in the working directory by default.

Directory-separated environments rely on duplicate Terraform code. This may be useful if you want to test changes in a development environment before promoting them to production. However, the directory structure runs the risk of creating drift between the environments over time. If you want to reconfigure a project with a single state file into directory-separated states, you must perform advanced state operations to move the resources.

After reorganizing your environments into directories, your file structure should look like the one below.

.├── assets│   ├── index.html├── prod│   ├── main.tf│   ├── variables.tf│   ├── terraform.tfstate│   └── terraform.tfvars└── dev   ├── main.tf   ├── variables.tf   ├── terraform.tfstate   └── terraform.tfvars

Create`prod` and`dev` directories

Create directories namedprod anddev.

$ mkdir prod&& mkdir dev

Move thedev.tf file to thedev directory, and rename it tomain.tf.

$ mv dev.tf dev/main.tf

Copy thevariables.tf,terraform.tfvars, andoutputs.tf files to thedev directory

$ cp outputs.tf terraform.tfvars variables.tf dev/

Your environment directories are now one step removed from theassets folder where your webapp lives. Open thedev/main.tf file in your text editor and edit the file to reflect this change by editing the file path in thecontent argument of theaws_s3_object resource with a/.. before theassets subdirectory.

dev/main.tf

 resource "aws_s3_object" "dev" {   acl          = "public-read"   key          = "index.html"   bucket       = aws_s3_bucket.dev.id-  content      = file("${path.module}/assets/index.html")+  content      = file("${path.module}/../assets/index.html")   content_type = "text/html" }

You will need to remove the references to theprod environment from yourdev configuration files.

First, opendev/outputs.tf in your text editor and remove the reference to theprod environment.

dev/outputs.tf

-output "prod_website_endpoint" {-  value = "http://${aws_s3_bucket_website_configuration.prod.website_endpoint}/index.html"-}

Next, opendev/variables.tf and remove the reference to theprod environment.

dev/variables.tf

-variable "prod_prefix" {-  description = "This is the environment where your webapp is deployed. qa, prod, or dev"-}

Finally, opendev/terraform.tfvars and remove the reference to theprod environment.

dev/terraform.tfvars

 region      = "us-east-2"-prod_prefix = "prod" dev_prefix  = "dev"

Create a`prod` directory

Renameprod.tf tomain.tf and move it to your production directory.

$ mv prod.tf prod/main.tf

Move thevariables.tf,terraform.tfvars, andoutputs.tf files to theprod directory.

$ mv outputs.tf terraform.tfvars variables.tf prod/

Repeat the steps you took in thedev directory, and uncomment out therandom_pet andprovider blocks inmain.tf.

First, openprod/main.tf and edit it to reflect new directory structure by adding/.. to the file path in thecontent argument of theaws_s3_object, before theassets subdirectory.

Next, remove the references to thedev environment fromprod/variables.tf,prod/outputs.tf, andprod/terraform.tfvars.

Finally, uncommentterraform block, theprovider block, and therandom_pet resource inprod/main.tf.

prod/main.tf

-/* terraform {   required_providers {     aws = {       source = "hashicorp/aws"       version = "~> 4.0.0"     }     random = {       source  = "hashicorp/random"       version = "~> 3.1.0"     }   } } provider "aws" {   region = var.region } resource "random_pet" "petname" {   length    = 3   separator = "-" }-*/

Tip

To fast-forward to this configuration, rungit checkout directories.

Deploy environments

To deploy, change directories into your development environment.

$ cd dev

This directory is new to Terraform, so you must initialize it.

$ terraform init

Run an apply for the development environment and enteryes when prompted to accept the changes.

$ terraform apply

You now have only one output from this deployment. Check your website endpoint in a browser.

Repeat these steps for your production environment.

$ cd ../prod

This directory is new to Terraform, so you must initialize it first.

$ terraform init

Run your apply for your production environment and enteryes when prompted to accept the changes. Check your website endpoint in a browser.

$ terraform apply

Now your development and production environments are in separate directories, each with their own configuration files and state.

Destroy infrastructure

Before moving on to the second approach to environment separation, destroy both thedev andprod resources.

$ terraform destroy

To learn about another method of environment separation, navigate to the"Workspaces" tab.

Workspace-separated environments use the same Terraform code but have different state files, which is useful if you want your environments to stay as similar to each other as possible, for example if you are providing development infrastructure to a team that wants to simulate running in production.

However, you must manage your workspaces in the CLI and be aware of the workspace you are working in to avoid accidentally performing operations on the wrong environment.

Update your Terraform directory

Note

If you ran the directory separation example, begin this section by removing your environment directories withrm -rf dev/ prod/ in your root directory. Then, switch branches by runninggit checkout file-separation in your terminal.

All Terraform configurations start out in thedefault workspace. Typeterraform workspace list to have Terraform print out the list of your workspaces with the currently selected one denoted by a*.

$ terraform workspace list   *  default

Before you create a new workspace, you need to update your configuration files so that both environments can use the same one. In your root directory, remove theprod.tf file.

$ rm prod.tf

Update your variable input file to remove references to the individual environments.

In your text editor, openvariables.tf and remove the environment references.

variables.tf

 variable "region" {   description = "This is the cloud hosting region where your webapp will be deployed." }-variable "dev_prefix" {+variable "prefix" {   description = "This is the environment where your webapp is deployed. qa, prod, or dev" }-variable "prod_prefix" {-  description = "This is the environment where your webapp is deployed. qa, prod, or dev"-}

Renamedev.tf tomain.tf.

$ mv dev.tf main.tf

Open this file in your text editor and replace the "dev" resource IDs and variables with the function of the resource itself. You are creating a generic configuration file that can apply to multiple environments.

main.tf

-resource "aws_s3_bucket" "dev" {+resource "aws_s3_bucket" "bucket" {-  bucket = "${var.dev_prefix}-${random_pet.petname.id}"+  bucket = "${var.prefix}-${random_pet.petname.id}"   force_destroy = true }-resource "aws_s3_bucket_website_configuration" "dev" {+resource "aws_s3_bucket_website_configuration" "bucket" {-  bucket = aws_s3_bucket.dev.id+  bucket = aws_s3_bucket.bucket.id   index_document {     suffix = "index.html"   }   error_document {     key = "error.html"   } }-resource "aws_s3_bucket_acl" "dev" {+resource "aws_s3_bucket_acl" "bucket" {-  bucket = aws_s3_bucket.dev.id+  bucket = aws_s3_bucket.bucket.id  acl = "public-read"}- resource "aws_s3_bucket_policy" "dev" {+ resource "aws_s3_bucket_policy" "bucket" {-   bucket = aws_s3_bucket.dev.id+   bucket = aws_s3_bucket.bucket.id   policy = <<EOF {     "Version": "2012-10-17",     "Statement": [         {             "Sid": "PublicReadGetObject",             "Effect": "Allow",             "Principal": "*",             "Action": [                 "s3:GetObject"             ],             "Resource": [-                "arn:aws:s3:::${aws_s3_bucket.dev.id}/*"+                "arn:aws:s3:::${aws_s3_bucket.bucket.id}/*"             ]         }     ] } EOF }-resource "aws_s3_object" "dev" {+resource "aws_s3_object" "webapp" {   acl          = "public-read"   key          = "index.html"-  bucket       = aws_s3_bucket.dev.id+  bucket       = aws_s3_bucket.bucket.id   content      = file("${path.module}/assets/index.html")   content_type = "text/html" }

Using workspaces organizes the resources in your state file by environments, so you only need one output value definition. Open youroutputs.tf file in your text editor and remove thedev environment reference in the output name. Changedev in the value tobucket.

outputs.tf

output "website_endpoint" {  value= "http://${aws_s3_bucket_website_configuration.bucket.website_endpoint}/index.html"}

Finally, replaceterraform.tfvars with aprod.tfvars file and adev.tfvars file to define your variables for each environment.

For yourdev workspace, copy theterraform.tfvars file to a newdev.tfvars file.

$ cp terraform.tfvars dev.tfvars

Edit the variable definitions in your text editor. For yourdev workspace, the prefix value should bedev.

dev.tfvars

region= "us-east-2"prefix= "dev"

Create a new.tfvars file for your production environment variables by renaming theterraform.tfvars file toprod.tfvars.

$ mv terraform.tfvars prod.tfvars

Updateprod.tfvars with yourprod prefix.

prod.tfvars

region= "us-east-2"prefix= "prod"

Now that you have a singlemain.tf file, initialize your directory to ensure your Terraform configuration is valid.

$ terraform init

Tip

To fast-forward to this configuration rungit checkout workspaces.

Create a`dev` workspace

Create a new workspace in the Terraform CLI with theworkspace command.

$ terraform workspace new dev

Terraform's output will confirm you created and switched to the workspace.

Created and switched to workspace "dev"!You're now on a new, empty workspace. Workspaces isolate their state,so if you run "terraform plan" Terraform will not see any existing statefor this configuration.

Any previous state files from yourdefault workspace are hidden from yourdev workspace, but your directory and file structure do not change.

Initialize the directory.

$ terraform init

Apply the configuration for your development environment in the new workspace, specifying thedev.tfvars file with the-var-file flag.

$ terraform apply -var-file=dev.tfvars

Terraform will create three resources and prompt you to confirm that you want to perform these actions in the workspace "dev".

## ...Do you want to perform these actions in workspace "dev"?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.  Enter a value:  ## ...

Enteryes and check your website endpoint in a browser.

Create a prod workspace

Create a new production workspace.

$ terraform workspace new prod

Terraform's output will confirm you created the workspace and are operating within that workspace.

Created and switched to workspace "prod"!You're now on a new, empty workspace. Workspaces isolate their state,so if you run "terraform plan" Terraform will not see any existing statefor this configuration.

Any previous state files from yourdev workspace are hidden from yourprod workspace, but your directory and file structure do not change.

You have a specificprod.tfvars file for your new workspace. Runterraform apply with the-var-file flag and reference the file. Enteryes when you are prompted to accept the changes and check your website endpoint in a browser.

$ terraform apply -var-file=prod.tfvars

Your output now contains only resources labeled "production" and your single website endpoint is prefixed withprod.

State storage in workspaces

When you use the default workspace with the local backend, yourterraform.tfstate file is stored in the root directory of your Terraform project. When you add additional workspaces your state location changes; Terraform internals manage and store state files in the directoryterraform.tfstate.d.

Your directory will look similar to the one below.

.├── README.md├── assets│   └── index.html├── dev.tfvars├── main.tf├── outputs.tf├── prod.tfvars├── terraform.tfstate.d│   ├── dev│   │   └── terraform.tfstate│   ├── prod│   │   └── terraform.tfstate├── terraform.tfvars└── variables.tf

Destroy your workspace deployments

To destroy your infrastructure in a multiple workspace deployment, you must select the intended workspace and runterraform destroy -var-file= with the.tfvars file that corresponds to your workspace.

Destroy the infrastructure in yourprod workspace, specifying theprod.tfvars file with the-var-file flag.

$ terraform destroy -var-file=prod.tfvars## ...Plan: 0 to add, 0 to change, 3 to destroy.Changes to Outputs:  - website_endpoint = "http://prod-definitely-resolved-ghoul.s3-website.us-east-2.amazonaws.com/index.html" -> nullDo you really want to destroy all resources in workspace "prod"?  Terraform will destroy all your managed infrastructure, as shown above.  There is no undo. Only 'yes' will be accepted to confirm.  Enter a value:

When you are sure you are running yourdestroy command in the correct workspace, enteryes to confirm the destroy plan.

Next, to destroy your development infrastructure, switch to yourdev workspace using theselect subcommand.

$ terraform workspaceselect dev

Runterraform destroy specifyingdev.tfvars with the-var-file flag.

$ terraform destroy -var-file=dev.tfvars

Next steps

In this exercise, you learned how to restructure a monolithic Terraform configuration that managed multiple environments. You separated those environments by creating different directories or workspaces, and state files for each. To learn more about how to organize your configuration, review the following resources:

Learn how touse and create modules to combat configuration drift.
Learn about howHCP Terraform eases state management and using Terraform as a team.
Learn how touse remote backends and migrate your configuration.

Add from the public registry

Create modules

This tutorial also appears in:

33 tutorials
Terraform Associate (003) Tutorials
Progress through these tutorials to prepare for the Terraform Associate (003) certification exam.
- Terraform

Movatterモバイル変換