DRY using Terraform

Do not repeat yourself using Terraform

Featured image

DRY?

In the world of Infrastructure as Code (IaC), the principle of DRY (Don’t Repeat Yourself) is crucial for maintaining efficient and manageable codebases. When it comes to Terraform, one of the most popular IaC tools, applying DRY principles can significantly improve your infrastructure management in a lot of ways. But what exactly does DRY mean in the context of Terraform, and why is it so important for you and most importantly, for your current job and company?

DRY in Terraform refers to the practice of minimizing code duplication and promoting reusability across your infrastructure configurations. This approach not only reduces the potential for errors but also enhances maintainability and scalability of your infrastructure code. As your infrastructure grows in complexity, adhering to DRY principles becomes increasingly vital for scaling in today’s cloud environments.

In this blog post, we’ll not explore the concept of DRY in Terraform as there are many other blogs that talk about this. What we will do is explore two tools that allows us to apply DRY principles in our beloved Terraform file; Terragrunt and Atmos.

Terragrunt

Terragrunt is a thin wrapper around Terraform that helps manage and work with complex infrastructure-as-code (IaC) configurations by solving some common challenges developers face when working with Terraform. It’s designed to make Terraform easier to use by providing additional features like code reuse, dependency management, and automated remote state management.

Terraform, while powerful, can become difficult to manage in larger, more complex infrastructure projects due to several challenges like:

  1. Duplicated Code: Terraform configurations often require similar code across different environments (e.g., staging, production). Managing these environments can lead to a lot of copy-pasting of the same code with minor modifications, which is hard to maintain.
  2. Remote State Management: Terraform requires state to track resources, and in larger teams, managing Terraform state files across multiple environments can be tricky.
  3. Dependency Management: Terraform doesn’t handle resource dependencies across different modules/environments well out-of-the-box. For example, if a networking module needs to be deployed before an application module, Terraform doesn’t provide a built-in solution for cross-module dependencies.
  4. Managing Multiple Environments/Regions: Deploying infrastructure to multiple environments (e.g., dev, staging, production) and regions can involve many configurations and make infrastructure drift hard to control.

Benefits of Using Terragrunt

1. DRY / Reusable Configurations

Terragrunt allows you to eliminate duplicate code across environments by enabling modular and reusable Terraform configurations. It lets you use a single Terraform module across multiple environments by passing in different configurations.

Example:

hcl
Copy code
# terragrunt.hcl for "production"
terraform {
  source = "../modules/my-app"
}

inputs = {
  environment = "prod"
  instance_type = "m5.large"
}

# terragrunt.hcl for "development"
terraform {
  source = "../modules/my-app"
}

inputs = {
  environment = "dev"
  instance_type = "t3.micro"
}

2. Automated Remote State Management

Managing Terraform state files can be tricky, especially when using remote state backends like S3 or GCS. Terragrunt simplifies this by automatically configuring remote state for you.

Example of remote state configuration in terragrunt.hcl:

hcl
Copy code
remote_state {
  backend = "s3"
  config = {
    bucket         = "my-terraform-state"
    key            = "envs/prod/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-lock"
  }
}

3. Dependency Management

Terragrunt allows you to define dependencies between different Terraform modules. For example, if you want to ensure that your network infrastructure is deployed before your application infrastructure, Terragrunt provides a built-in way to handle these dependencies.

Example:

hcl
Copy code
dependency "vpc" {
  config_path = "../vpc"
}

inputs = {
  vpc_id = dependency.vpc.outputs.vpc_id
}

4. Managing Multiple Environments/Regions Easily

Terragrunt simplifies managing multiple environments and regions by allowing you to keep the configurations for environments and regions in one place and reference shared modules. This makes managing environments like dev, staging, and prod much easier, and ensures consistency.

For example, you can define an infrastructure layout like this:

bash
Copy code
/live
  └── /dev
      └── terragrunt.hcl
  └── /prod
      └── terragrunt.hcl

5. Built-In Workflows for apply-all, destroy-all

Terragrunt provides higher-level commands like apply-all and destroy-all, which make it easy to apply or destroy Terraform configurations across multiple modules and environments in one go.

6. Easier Multi-Account or Multi-Region Setups

When managing infrastructure across different AWS accounts or regions, Terragrunt makes it easier to manage credentials and configuration by allowing you to reuse a single configuration file with different variables (e.g., different AWS account credentials).

Use Cases for Terragrunt

Atmos

Atmos is a tool developed by CloudPosse to enhance the workflow for managing infrastructure with Terraform and Helm. It is designed to simplify and standardize the configuration of complex environments by introducing a consistent, modular, and scalable approach.

Benefits of using Atmos:

  1. Hierarchical Configuration Management: Atmos allows you to define infrastructure and environment configurations hierarchically. You can share configurations between environments (like dev, staging, and prod), while allowing specific overrides.

    Example:

     yaml
     Copy code
     # catalog/terraform/stacks/dev.yaml
     global:
       region: us-west-1
       account_id: "123456789012"
     components:
       terraform:
         vpc:
           vars:
             cidr_block: "10.0.0.0/16"
        
    
     yaml
     Copy code
     # catalog/terraform/stacks/prod.yaml
     global:
       region: us-east-1
       account_id: "987654321098"
     components:
       terraform:
         vpc:
           vars:
             cidr_block: "172.16.0.0/16"
        
    

    In this example, the vpc configuration for prod overrides the region and CIDR block while sharing common components defined globally.

  2. Modular Architecture: Atmos promotes reusable modules across environments, reducing the need to redefine infrastructure repeatedly.

    Example: You can define a shared vpc module used across all environments:

     hcl
     Copy code
     # modules/vpc/main.tf
     module "vpc" {
       source = "terraform-aws-modules/vpc/aws"
       name   = var.vpc_name
       cidr   = var.cidr_block
     }
        
    

    Then reference this module in each environment’s configuration:

     yaml
     Copy code
     # stacks/dev.yaml
     components:
       terraform:
         vpc:
           vars:
             vpc_name: "dev-vpc"
             cidr_block: "10.0.0.0/16"
        
    
  3. Layering System: Atmos layers your infrastructure components (networking, apps, security) across environments, simplifying management.

    Example:

     yaml
     Copy code
     # stacks/dev.yaml
     components:
       terraform:
         vpc:
           vars:
             vpc_name: "dev-vpc"
         app:
           vars:
             app_name: "my-app-dev"
        
    
     yaml
     Copy code
     # stacks/prod.yaml
     components:
       terraform:
         vpc:
           vars:
             vpc_name: "prod-vpc"
         app:
           vars:
             app_name: "my-app-prod"
        
    

    Here, both environments share the same module but apply different variables per layer (vpc, app).

  4. YAML-Based Configuration: Atmos uses YAML for defining infrastructure stacks, making it easier to read and maintain.

    Example:

     yaml
     Copy code
     # stacks/dev.yaml
     global:
       region: us-west-1
     components:
       terraform:
         eks:
           vars:
             cluster_name: "dev-cluster"
             node_count: 3
        
    
  5. Integrated with Terraform and Helm: Atmos integrates with Terraform for infrastructure provisioning and Helm for Kubernetes, allowing you to handle both through a unified CLI.

    Example: Run Terraform plan for the eks component:

     bash
     Copy code
     atmos terraform plan eks
        
    

    Run Helm upgrade for a Kubernetes service:

     bash
     Copy code
     atmos helm upgrade service-name
        
    
  6. Support for Multi-Environment Workflows: Atmos helps manage multiple environments and regions without repeating code by automatically selecting configurations based on context.

    Example: To deploy infrastructure to the prod environment:

     bash
     Copy code
     atmos terraform apply eks --stack=prod
        
    
  7. Stack Configuration: Atmos introduces “stacks” to define infrastructure across environments in a repeatable manner.

    Example:

     yaml
     Copy code
     # catalog/terraform/stacks/prod.yaml
     global:
       region: us-east-1
       account_id: "987654321098"
     components:
       terraform:
         vpc:
           vars:
             cidr_block: "172.16.0.0/16"
         eks:
           vars:
             cluster_name: "prod-cluster"
        
    

    This stack defines vpc and eks configurations for the prod environment.

  8. Terraform Backend and Workspaces Management: Atmos simplifies managing Terraform backends and workspaces across environments. The backend configuration is handled automatically based on the stack.

    Example:

     yaml
     Copy code
     # stacks/prod.yaml
     terraform:
       backend:
         s3:
           bucket: "my-tf-state-prod"
           key: "terraform/state"
           region: "us-east-1"
        
    
  9. CLI for Orchestration: Atmos provides a powerful CLI to run Terraform and Helm commands while automatically applying the right configurations.

    Example: To deploy to prod:

     bash
     Copy code
     atmos terraform apply eks --stack=prod
        
    

    To destroy a resource:

     bash
     Copy code
     atmos terraform destroy vpc --stack=dev
        
    
  10. Standardization Across Teams: Atmos enforces a consistent structure for Terraform and Helm configurations, reducing the risk of errors and making it easier to onboard new team members.

Example Structure:

bash
Copy code
catalog/
├── terraform/
│   ├── modules/
│   └── stacks/
└── helmfile/
    ├── charts/
    └── releases/

Conclusion

Both Terragrunt and Atmos offer valuable enhancements to infrastructure management with Terraform. Terragrunt focuses on code reuse, dependency management, and remote state management, making it particularly useful for complex, multi-environment, and multi-region projects. It helps enforce consistency while reducing duplication in Terraform configurations.

Atmos, on the other hand, brings additional value by integrating Terraform and Helm management. It simplifies configuration management through its layering and stack systems, promotes modular and reusable setups, and provides a standardized workflow. This approach is especially beneficial for teams managing complex multi-environment infrastructures.

Both tools aim to help teams scale infrastructure management efficiently and consistently across multiple environments and regions. Whether you choose Terragrunt or Atmos depends on your specific needs, but both can significantly improve your infrastructure management workflows, especially in large-scale, complex environments.

References

Build On!