13 min to read
DRY using Terraform
Do not repeat yourself using Terraform
DRY?
In the world of Infrastructure as Code (IaC), the principle of DRY (Don’t Repeat Yourself) is crucial for maintaining efficient and manageable codebases. When it comes to Terraform, one of the most popular IaC tools, applying DRY principles can significantly improve your infrastructure management in a lot of ways. But what exactly does DRY mean in the context of Terraform, and why is it so important for you and most importantly, for your current job and company?
DRY in Terraform refers to the practice of minimizing code duplication and promoting reusability across your infrastructure configurations. This approach not only reduces the potential for errors but also enhances maintainability and scalability of your infrastructure code. As your infrastructure grows in complexity, adhering to DRY principles becomes increasingly vital for scaling in today’s cloud environments.
In this blog post, we’ll not explore the concept of DRY in Terraform as there are many other blogs that talk about this. What we will do is explore two tools that allows us to apply DRY principles in our beloved Terraform file; Terragrunt and Atmos.
Terragrunt
Terragrunt is a thin wrapper around Terraform that helps manage and work with complex infrastructure-as-code (IaC) configurations by solving some common challenges developers face when working with Terraform. It’s designed to make Terraform easier to use by providing additional features like code reuse, dependency management, and automated remote state management.
Terraform, while powerful, can become difficult to manage in larger, more complex infrastructure projects due to several challenges like:
- Duplicated Code: Terraform configurations often require similar code across different environments (e.g., staging, production). Managing these environments can lead to a lot of copy-pasting of the same code with minor modifications, which is hard to maintain.
- Remote State Management: Terraform requires state to track resources, and in larger teams, managing Terraform state files across multiple environments can be tricky.
- Dependency Management: Terraform doesn’t handle resource dependencies across different modules/environments well out-of-the-box. For example, if a networking module needs to be deployed before an application module, Terraform doesn’t provide a built-in solution for cross-module dependencies.
- Managing Multiple Environments/Regions: Deploying infrastructure to multiple environments (e.g., dev, staging, production) and regions can involve many configurations and make infrastructure drift hard to control.
Benefits of Using Terragrunt
1. DRY / Reusable Configurations
Terragrunt allows you to eliminate duplicate code across environments by enabling modular and reusable Terraform configurations. It lets you use a single Terraform module across multiple environments by passing in different configurations.
- Without Terragrunt: You often end up copying and pasting the same Terraform code in different files for different environments, which can be hard to maintain.
- With Terragrunt: You can create a base Terraform module and manage different environments (e.g., dev, prod) by simply configuring input variables in Terragrunt’s
terragrunt.hcl
files. This reduces duplication and increases maintainability.
Example:
hcl
Copy code
# terragrunt.hcl for "production"
terraform {
source = "../modules/my-app"
}
inputs = {
environment = "prod"
instance_type = "m5.large"
}
# terragrunt.hcl for "development"
terraform {
source = "../modules/my-app"
}
inputs = {
environment = "dev"
instance_type = "t3.micro"
}
2. Automated Remote State Management
Managing Terraform state files can be tricky, especially when using remote state backends like S3 or GCS. Terragrunt simplifies this by automatically configuring remote state for you.
- Without Terragrunt: You manually configure and update remote state settings in each Terraform configuration.
- With Terragrunt: You can define remote state settings once in Terragrunt and reuse it across all environments/modules. Terragrunt will automatically handle the configuration and lock state files to prevent conflicts.
Example of remote state configuration in terragrunt.hcl
:
hcl
Copy code
remote_state {
backend = "s3"
config = {
bucket = "my-terraform-state"
key = "envs/prod/terraform.tfstate"
region = "us-west-2"
encrypt = true
dynamodb_table = "terraform-lock"
}
}
3. Dependency Management
Terragrunt allows you to define dependencies between different Terraform modules. For example, if you want to ensure that your network infrastructure is deployed before your application infrastructure, Terragrunt provides a built-in way to handle these dependencies.
- Without Terragrunt: Terraform does not handle cross-module or cross-environment dependencies. You need to manually ensure that dependencies are applied in the correct order.
- With Terragrunt: You can explicitly define dependencies between modules using Terragrunt, and it will manage the execution order.
Example:
hcl
Copy code
dependency "vpc" {
config_path = "../vpc"
}
inputs = {
vpc_id = dependency.vpc.outputs.vpc_id
}
4. Managing Multiple Environments/Regions Easily
Terragrunt simplifies managing multiple environments and regions by allowing you to keep the configurations for environments and regions in one place and reference shared modules. This makes managing environments like dev
, staging
, and prod
much easier, and ensures consistency.
- Without Terragrunt: You would have separate Terraform configurations for each environment, which can lead to code duplication and potential drift between environments.
- With Terragrunt: You can maintain a clean, consistent structure for environments, with separate
terragrunt.hcl
files for each environment that specify how to configure the underlying shared modules.
For example, you can define an infrastructure layout like this:
bash
Copy code
/live
└── /dev
└── terragrunt.hcl
└── /prod
└── terragrunt.hcl
5. Built-In Workflows for apply-all
, destroy-all
Terragrunt provides higher-level commands like apply-all
and destroy-all
, which make it easy to apply or destroy Terraform configurations across multiple modules and environments in one go.
- Without Terragrunt: You need to manually run
terraform apply
orterraform destroy
for each module or environment. - With Terragrunt: You can use a single command (
terragrunt apply-all
orterragrunt destroy-all
) to apply or destroy all configurations across multiple modules and environments.
6. Easier Multi-Account or Multi-Region Setups
When managing infrastructure across different AWS accounts or regions, Terragrunt makes it easier to manage credentials and configuration by allowing you to reuse a single configuration file with different variables (e.g., different AWS account credentials).
- Without Terragrunt: You may have to manually configure provider blocks for each account or region.
- With Terragrunt: You can configure a base
terragrunt.hcl
that handles different accounts or regions by passing in different variables.
Use Cases for Terragrunt
- Large teams with multiple environments (dev, staging, prod): It simplifies managing the differences between environments without duplicating code.
- Infrastructure spread across multiple regions/accounts: Helps reduce complexity by handling remote state and dependencies across environments and accounts.
- Cross-module dependencies: Makes managing the order of module execution much simpler when some modules depend on others.
- Modular, DRY infrastructure: Ideal for teams that want to reuse Terraform code across environments.
Atmos
Atmos is a tool developed by CloudPosse to enhance the workflow for managing infrastructure with Terraform and Helm. It is designed to simplify and standardize the configuration of complex environments by introducing a consistent, modular, and scalable approach.
Benefits of using Atmos:
-
Hierarchical Configuration Management: Atmos allows you to define infrastructure and environment configurations hierarchically. You can share configurations between environments (like
dev
,staging
, andprod
), while allowing specific overrides.Example:
yaml Copy code # catalog/terraform/stacks/dev.yaml global: region: us-west-1 account_id: "123456789012" components: terraform: vpc: vars: cidr_block: "10.0.0.0/16"
yaml Copy code # catalog/terraform/stacks/prod.yaml global: region: us-east-1 account_id: "987654321098" components: terraform: vpc: vars: cidr_block: "172.16.0.0/16"
In this example, the
vpc
configuration forprod
overrides the region and CIDR block while sharing common components defined globally. -
Modular Architecture: Atmos promotes reusable modules across environments, reducing the need to redefine infrastructure repeatedly.
Example: You can define a shared
vpc
module used across all environments:hcl Copy code # modules/vpc/main.tf module "vpc" { source = "terraform-aws-modules/vpc/aws" name = var.vpc_name cidr = var.cidr_block }
Then reference this module in each environment’s configuration:
yaml Copy code # stacks/dev.yaml components: terraform: vpc: vars: vpc_name: "dev-vpc" cidr_block: "10.0.0.0/16"
-
Layering System: Atmos layers your infrastructure components (networking, apps, security) across environments, simplifying management.
Example:
yaml Copy code # stacks/dev.yaml components: terraform: vpc: vars: vpc_name: "dev-vpc" app: vars: app_name: "my-app-dev"
yaml Copy code # stacks/prod.yaml components: terraform: vpc: vars: vpc_name: "prod-vpc" app: vars: app_name: "my-app-prod"
Here, both environments share the same module but apply different variables per layer (
vpc
,app
). -
YAML-Based Configuration: Atmos uses YAML for defining infrastructure stacks, making it easier to read and maintain.
Example:
yaml Copy code # stacks/dev.yaml global: region: us-west-1 components: terraform: eks: vars: cluster_name: "dev-cluster" node_count: 3
-
Integrated with Terraform and Helm: Atmos integrates with Terraform for infrastructure provisioning and Helm for Kubernetes, allowing you to handle both through a unified CLI.
Example: Run Terraform plan for the
eks
component:bash Copy code atmos terraform plan eks
Run Helm upgrade for a Kubernetes service:
bash Copy code atmos helm upgrade service-name
-
Support for Multi-Environment Workflows: Atmos helps manage multiple environments and regions without repeating code by automatically selecting configurations based on context.
Example: To deploy infrastructure to the
prod
environment:bash Copy code atmos terraform apply eks --stack=prod
-
Stack Configuration: Atmos introduces “stacks” to define infrastructure across environments in a repeatable manner.
Example:
yaml Copy code # catalog/terraform/stacks/prod.yaml global: region: us-east-1 account_id: "987654321098" components: terraform: vpc: vars: cidr_block: "172.16.0.0/16" eks: vars: cluster_name: "prod-cluster"
This stack defines
vpc
andeks
configurations for theprod
environment. -
Terraform Backend and Workspaces Management: Atmos simplifies managing Terraform backends and workspaces across environments. The backend configuration is handled automatically based on the stack.
Example:
yaml Copy code # stacks/prod.yaml terraform: backend: s3: bucket: "my-tf-state-prod" key: "terraform/state" region: "us-east-1"
-
CLI for Orchestration: Atmos provides a powerful CLI to run Terraform and Helm commands while automatically applying the right configurations.
Example: To deploy to
prod
:bash Copy code atmos terraform apply eks --stack=prod
To destroy a resource:
bash Copy code atmos terraform destroy vpc --stack=dev
-
Standardization Across Teams: Atmos enforces a consistent structure for Terraform and Helm configurations, reducing the risk of errors and making it easier to onboard new team members.
Example Structure:
bash
Copy code
catalog/
├── terraform/
│ ├── modules/
│ └── stacks/
└── helmfile/
├── charts/
└── releases/
Conclusion
Both Terragrunt and Atmos offer valuable enhancements to infrastructure management with Terraform. Terragrunt focuses on code reuse, dependency management, and remote state management, making it particularly useful for complex, multi-environment, and multi-region projects. It helps enforce consistency while reducing duplication in Terraform configurations.
Atmos, on the other hand, brings additional value by integrating Terraform and Helm management. It simplifies configuration management through its layering and stack systems, promotes modular and reusable setups, and provides a standardized workflow. This approach is especially beneficial for teams managing complex multi-environment infrastructures.
Both tools aim to help teams scale infrastructure management efficiently and consistently across multiple environments and regions. Whether you choose Terragrunt or Atmos depends on your specific needs, but both can significantly improve your infrastructure management workflows, especially in large-scale, complex environments.
References
Build On!