6 min to read
Running Terraform in the cloud!
A post about Terraform + AWS Spot Instances + Lambdas and Making your life easier. This post starts by explaining the different technologies that are needed in orded to automate the following process:
Launch AWS Spot Instances automatically that are available only on business hours.
Prologue
Pets vs Catle
In February 2012, Randy Bias gave an impactful talk on architectures for open and scalable clouds. In his presentation, he established the pets versus cattle meme:
- With the pets approach to infrastructure, you treat the machines as individuals. You give each (virtual) machine a name, and applications are statically allocated to machines. For example, db-prod-2 is one of the production servers for a database. The apps are manually deployed, and when a machine gets ill you nurse it back to health and manually redeploy the app it ran onto another machine. This approach is generally considered to be the dominant paradigm of a previous (non–cloud native) era.
- With the cattle approach to infrastructure, your machines are anonymous, they are all identical (modulo hardware upgrades), they have numbers rather than names, and apps are automatically deployed onto any and each of the machines. When one of the machines gets ill, you don’t worry about it immediately; you replace it—or parts of it, such as a faulty hard disk drive— when you want and not when things break.
Go Cattle!
The beautiful thing about applying the cattle approach to infrastructure is that it allows you to scale out on commodity hardware.
It gives you elasticity with the implication of hybrid cloud capabilities. This is a fancy way of saying that you can have parts of your deployments on premises and burst into the public cloud—using services provided by the likes of Amazon, Microsoft, and Google, or the infrastructure-as-a-service (IaaS) offerings of different provides like VMware—if and when you need to.
Most importantly, from an operator’s point of view, the cattle approach allows you to get a decent night’s sleep, as you’re no longer paged at 3 a.m. just to replace a broken hard disk drive or to relaunch a hanging app on a different server, as you would have done with your pets.
Now that you know about pets versus cattle, you are ready to have a look at Terraform, Spot Instances and Lambdas.
Terraform
According to their official website, HashiCorp’s Terraform enables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.
Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.
The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.
AWS Spot Instances
In a nutshell Spot instances are spare compute capacity in the AWS cloud available to you at steep discounts compared to On-Demand prices. EC2 Spot enables you to optimize your costs on the AWS cloud and scale your application’s throughput up to 10X for the same budget. By simply selecting Spot when launching EC2 instances, you can save up-to 90% on On-Demand prices.
The only difference between On-Demand instances and Spot Instances is that Spot instances can be interrupted by EC2 with two minutes of notification when EC2 needs the capacity back. You can use EC2 Spot for various fault-tolerant and flexible applications, such as test & development environments, stateless web servers, image rendering, video transcoding, and to run analytics, machine learning and high-performance computing (HPC) workloads.
AWS Lambda
Basically, AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running.
With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.
AWS CloudWatch Events
Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams. CloudWatch Events becomes aware of operational changes as they occur. CloudWatch Events responds to these operational changes and takes corrective action as necessary, by sending messages to respond to the environment, activating functions, making changes, and capturing state information.
The Problem
Launch AWS Spot Instances automatically that are available only on business hours. Thats the business requeriment as we need a lot of compute resources in order to process specific workloads that are managed by an specific team in the company only on business hours. At the same time we want to enable and “enhance” the cloud adoption, by this I mean, we don’t want to spend lots of money when using the cloud, we jut need the job done and in the best cheapest way.
Welcome Terraform, AWS Spot Instances, AWS Lambda and AWS CloudWatch Events. This technologies/services will enable us to do what business asked us to do. Let me show you how.
The Solution
Stay tunned for the part 2!
