CDK for Terraform with Python and its Operational Experience at ShopStyle

Learn how ShopStyle is using Python to manage their AWS infrastructure with the help of the CDK for Terraform.

»Transcript

Hi, my name is Charles McLaughlin, and I’m presenting Getting Started with the Cloud Development Kit for Terraform. First of all, thanks to HashiCorp, everyone that helped organize the conference, and everyone watching. 

»Introduction

About me. Let’s get to know each other a bit. I have over 15 years of experience in the industry, and I’ve always teetered on the line between systems and development. In the early years — for me — I was just excited to get things working. Now I’m more interested in refining my skills, reflecting, and getting it right.

While I’m grateful not to be on the front lines of the pandemic and I can safely work from home, our industry can certainly be frustrating at times. I find it difficult to come back to projects a month later without tests and best practices. Dealing with crufty code and yak shaving is no fun. So, when tools like CDK for Terraform come along, it makes work more enjoyable and fun.

I’ve been using HashiCorp Terraform for a while now, and I was an early adopter of the CDK for Terraform. I was quick to jump in and try it out. So far, I have over 60 stacks, and it’s been awesome. I work for ShopStyle. We’re part of the Rakuten family, and it’s a great place to work. If you’re interested in technology like this, please get in touch with me because I’m hiring DevOps engineers.

»CDKTF in a Nutshell

I’d like to give you an introduction before we dive into details. What is the CDK for Terraform? Under the hood, it uses a library called jsii, which was invented by Amazon for its CloudFormation-based CDK. 

CDKTF is written in TypeScript and runs under Node.js behind the scenes. But I’ll be showing you examples in Python — and despite the underlying magic — you can really ignore all of that and focus on your business’ needs.

The CDK project has been out for a little over a year, and it’s been stable and awesome for me. The community in HashiCorp has been very inviting. It’s great to be involved in such an awesome open source project. Behind the scenes, I mentioned it runs a conversion — you’re converting from your favorite programming language into HCL. I’ll go into that in more detail. 

In this presentation, I’ll be going over a couple of examples and showing some nice patterns that I think are useful. I have an example available on GitHub with some Python code that you can check out. However, these ideas are applicable to other programming languages as well. One particular pattern t I think is very useful is web developers writing apps in TypeScript can stick to their comfort zone and start to follow best practices using infrastructure as code. 

»What Problem Are We Solving? 

If you’re just getting started with Terraform, you should know that it’s the proven infrastructure as code leader. It’s cloud-agnostic, and there are tons of providers available, so you can use it for all sorts of different use cases — and it’s fast.

Why would you want to use the CDK for Terraform? It allows you to use your favorite programming language compared to Terraform’s HCL format, which gets the job done, but it can be a little difficult to read and come back to. 

Programming in your favorite programming language also allows you to follow best practices such as linting and unit tests and also allows easier integration. For instance, you can use Boto3 in Python or other AWS-SDKs directly in the Cloud Development Kit for Terraform. 

You could think about how that allows much more freedom and possibility. For instance, it might be difficult to build out a self-service dashboard for cloud provisioning in HCL, but now you have all the conveniences of a fully functional programming language. 

»Example Project Structure 

It’s not always easy to get started with new technology, but the CDKTF CLI makes it easy with templates for a variety of programming languages that are built-in. I’ve decided to structure my repo like this because you obviously don’t want to waste a lot of time copying and pasting code, and you don’t want a large monolithic stack with all of your infrastructure in one place.

Those design patterns are applicable to both HCL and the CDK for Terraform, but the CDK for Terraform makes it even easier. That could be a talk unto itself. We could get into writing tests and all that stuff. But let me add one caveat and point out that some of the solutions and the patterns that I’ve implemented here have already been implemented by the CDK for Terraform project — such as remote templates. In this example, I have some templates right here in the repository. I’m also going out of my way to show you that everything can be done in Python, and you don’t have to dive into a lot of the details of these conversion layers 

In this example, I also have a stacks directory and a lib directory. The lib directory in this example is for reusable Python modules. That’s where I have things such as an expanded Amazon S3 module that blocks public access by default — and that could be built upon for CloudFront distribution with static website hosting, for instance. 

I have a couple examples for you. We’re going to start with a simple one to demonstrate the elegance of the Cloud Development Kit for Terraform and then work our way up to another example that’s more advanced.

First, we’re going to set up an S3 bucket and DynamoDB table for Terraform state management and locking. The Cloud Development Kit for Terraform CLI can create your projects with automatic integration with Terraform Cloud. But I think this is a great example to get started with because it shows some basic patterns creating simple AWS resources — and we’ll be able to build upon this example. 

I already have a Python virtual environment installed and activated with all the dependencies for this demo, so we’ll be able to focus on the code here. I’ll start by seeding into the stacks directory, where I have two examples. 

»The CDK State Example

I’m going to open up main in my editor, and we’ll dive into the code. For starters, I have a bunch of imports. I’m importing a class for the Terraform AWS provider itself and classes to build out the S3 bucket and DynamoDB table. 

I also have some CDK for Terraform imports that allow me to build out a stack and an application, which we’ll cover in more detail shortly. I’m also importing a string from another class, which allows me to reuse the name of the S3 bucket and DynamoDB table in a few places without repeating myself. This is where it really starts to get interesting. Here you can see I have a class that’s based on Terraform stack. In the constructor, we initialize the Terraform AWS provider, set a simple class variable — then we create the S3 bucket, block public access to the S3 bucket, and create a DynamoDB table. It’s all pretty simple. In the main entry point, we instantiate the CDKTF application and the stack, and then we synthesize the stack from Python to HCL. 

Let me show you what that looks like when we run it. Let’s take a quick look at the synthesized output. This is HCL-compatible JSON. I may sound like a broken record here, but the CDKTF CLI could deploy this for you. But since I’ve been using Terraform for a while, I’m going to run this myself.

Wow. That’s fast. Terraform never ceases to amaze me. There you have it. We have a super simple stack that sets up Terraform state management for future deployments. 

»How To Use Inheritance for Stack Setup

The last example showed how quick and easy it is to get started with CDK for Terraform, but that didn’t show you anything you couldn’t already do with HCL. Here’s a more advanced example. This shows how to use inheritance for stack setup. It allows you to reduce and reuse code. I’ll also demonstrate how you can move code out of stacks into reusable libraries to avoid repetition. Of course, that’s possible with Terraform modules, but I found it’s even easier with CDK for Terraform. That’s also a great place for you to write tests.

As I previously mentioned, in the root of my example repository, I have a lib directory. That’s where I find it useful to store reusable Python code and modules that can be used as building blocks. I‘d like to show you my example CDKTF env module. Here you can see we’re importing the example stack. We also have a couple new imports, such as Terraform output and example S3 bucket class, which allows us to create S3 buckets and reuse that code. But the interesting thing here is that my stack class inherits from the example stack class. That allows us to automatically have data and environment settings built-in. 

Here you can see I’m using that in the S3 bucket name. This code goes on to create a static website using S3, creating the bucket and parsing in that static website configuration. After that is created, Terraform prints the website URL. 

Let’s run this code and see what it looks like. The CDKTF CLI can actually deploy this for you, but again, since I’ve had a few years of Terraform experience, I wrote a simple script to do this deployment. Let’s take a look. And it’s done. Let’s take a quick look at the site to make sure it’s working. There it is. That shows you just how easy it is. 

I hope you’ve enjoyed my examples. I’ve shown you how quick and easy it is to get started with the Cloud Development Kit for Terraform, and I’ve shown you a more advanced example that just scratches the surface of some of the possibilities. I hope you’ll consider it for your future projects.