Terraform Cloud Components

[00:00:00]
>> So this is my beautiful wireframe. It looks daunting, but I promise you it is not, it's just very modularized. And it helped to understand the flow of automation and essentially from top down the dependencies that are required to get to the bottom. So we've talked about automation providers, we've talked about automation tools.

[00:00:31]
We've talked about repository, source code providers and networks, and we've talked about a lot, right? How do you merge all that together? How do you blend all of that together into a way that it makes sense, that it feels easy to manage, and it's something that you can scale on as well, right?

[00:00:56]
And so what we're trying to do is we're trying to take the LEGOs approach. We're trying to say, okay, well, what are the components that I need that change or affect the different parts of my infrastructure, right? If I have networks, I need to figure out how to create networks.

[00:01:17]
If I have clusters, I need to figure out how to create clusters, right, things like that. And so where we start first is all the way at the top, Terraform Cloud. And again, this is why we wanna make accounts for Terraform Cloud because this is where we're actually gonna begin.

[00:01:33]
And the reason for that is because this is where all of our state's gonna go, right? So going back to the whole Pulumi and other things like that. There's a reason why Terraform Cloud, again, is so nice, and it's out of the box solutions. I don't have to set up Terraform Cloud, really, I can just start working with it.

[00:01:54]
And so in this case, that's where we start, right? The next thing that we do is we solve the next problem we talked about, permissions. Permissions in access control, right? So we don't immediately start saying, okay, let's create all of our resources and automation. We say, okay, we have two teams.

[00:02:12]
We have our DevOps team, this is the isolation and the scope of which that they maintain, right? And then here's the product team. And I hope you see what I meant by what you give developers access to [LAUGH] and what you give DevOps access to, right? And you can even see, DevOps even still touches things that developers do.

[00:02:34]
But developers in the product teams have access to manage their services, we manage everything else. And so our teams are really important, right? That's the next thing that we provision. Then in the third layer, we have what are called our automation repositories, right? And so these are the repositories that we will be building.

[00:02:57]
There's one, two, three, four, five, I think there's one more, six in total. But the five infrastructure components that we're gonna be building are, one, at least should be very straightforward, and two, reusable across any cloud. Okay, so I'm not saying that the code inside is reusable, I'm saying that the modularity and the way that we build these components is.

[00:03:19]
Meaning that a provider network should mean that you should have a AWS network automation repo. You should have a GCP repo for automation, right, for network, so forth and so on. And so with that mentality, the first thing we do is we actually create one for tfe, which is Terraform Enterprise or Terraform Cloud.

[00:03:39]
Now, why do we do that and why don't we go to source code? Can anyone guess why we might make a repository for Terraform Cloud's automation first? So the main reason is so that we don't have to redo it later, [LAUGH] right? If you start provisioning everything with automation in the beginning, then everything is automated, right?

[00:04:03]
And so before we even go into Terraform Cloud and start creating teams manually, which is something you might think to do, we ask ourselves the question. Can we automate this process actually, too? Could I actually make teams in Terraform and have it manage that way? And so that's what we do, we create our first repository, which is our terraform-tfe repository.

[00:04:27]
And the whole sole purpose of this repo is to just go back up to Terraform Cloud and provision all the things for us, right? So what we actually do is we create a Terraform Cloud account, right? And then we set up our first workspace to make it so that it can essentially manage itself, right?

[00:04:46]
And then what we do is instead of coming here to make our changes, right, GitOps, we come here, right? So if I wanna add a new team to Terraform Cloud, what would I do in a GitOps approach?
>> Just add them to the base repo.
>> Yep, and then do what?

[00:05:06]
>> Provision access, I guess.
>> Yeah, merge and then let it run, yeah. So the idea is that if I went here, I don't know who did it, I don't know when they added the team, I don't know how they did it. You know what I mean? There's no real auditable track here.

[00:05:20]
But if we create our Terraform Cloud account and then we just say, okay, everything's automated, we lean into that source of truth right at the gate. And we say not only is all of our other stuff managed through automation, but also our Terraform Cloud automation is actually in the same boat, right?

[00:05:38]
So that's where we start. Once we have automation for this, we can create teams in Terraform Cloud, and we can create the next thing, which is called workspaces, okay? So workspaces are essentially your automation pipelines for the specific component that you're trying to build, right? So if I was to say, I want my automation for Terraform Cloud to be managed by this repository, then I probably need a workspace for that repository to be tracked, right?

[00:06:13]
And that's the real takeaway here, is each one of these tracks, as you see down below, its own repo as well, right? And then that's the lowest part of it, which is once we set up our workspaces, we tell our workspaces to go out to GitHub and manage and sync them from the repo that we're using.

[00:06:36]
So I'm gonna ask some questions here. Actually, let me get through this first and then I'll ask my questions. So again, I've got my Terraform automation now, right? So my next question actually would be, where would I go if I wanted to make a new workspace? Would I go here?

[00:06:59]
Go here? Yes, yeah, I would go here, right? Because again, although this is what we set up, Terraform Cloud, this is now the source of truth, meaning technically we actually go here. We go down to the repo, we follow our normal GitHub's procedures. And then once we merge, it goes back up, and then it runs and runs against Terraform Cloud like this, and that's the actual flow.

[00:07:22]
So we make our changes here, goes back up, and then this workspace goes out to Terraform Cloud, creates the DevOps team, the product team. And then because we have this cycle now right here, right, we can now build other things. So then we create our GitHub workspace, we create our provider network workspace, our provider cluster workspace, and our product service workspace.

[00:07:48]
So my next question is, where would I want to go if I wanted to update GitHub repos? I wanna add a new GitHub repo, where would I go?
>> Same place.
>> Same place, here? Here, no. So technically, the only thing that this manages is Terraform Cloud resources.

[00:08:16]
This is where we start separating the automation from each other. And so what we say is we say, here everything is managed by Terraform Cloud, and here everything is managed for GitHub. Now, if you have a scenario where you wanna create a workspace that tracks a repo, do you have to make one change or two?

[00:08:38]
What do you think? So I wanna create a workspace and a repo with automation, where do you guys think I would go to do that? We said that we create workspaces in which repo?
>> In the first one, so you have to make it in two places.
>> Exactly, yes, you would have to make it in two places.

[00:08:58]
You would first make it in terraform-tfe to create the workspace and automation, get that approved, right? That goes out, creates your workspace. And then you go to GitHub and then you create your repository to then get tracked for that workspace, or you could do it any other way.

[00:09:13]
You could create the repository first and then have the workspace track the repo, right? This is a decision. The reason why I make this decision is because I don't want automation to dynamically create and delete my repositories for me. Can anyone guess why that might be terrifying? [LAUGH] Basically, you just make sure you don't lose any code.

[00:09:39]
[LAUGH] If you bind your repositories to other state, they're gonna go and disappear essentially with that state. And so if somebody accidentally upstate something somewhere else without realizing downstream, that's gonna delete a repo, that's a bit terrifying. And so at Hippo, we choosingly separate those processes just so that APIs cannot delete repositories.

[00:10:09]
They can be archived, but they cannot be deleted. And so yeah, in this case, what you would do is you would go to terraform-tfe to create your workspace, and then you go to terraform-github to create your repository. So we talked a bit about multi-cloud as well, right? And the next two repos really are about that concept of, okay, so we have a repo that focuses on a single problem, which is Terraform Cloud.

[00:10:39]
And we have another repo that focuses on a single problem, which is GitHub, right? Well, now we need to focus on a specific problem for a specific provider, right? But we don't wanna make it so we can just work with that provider, we wanna make it so that if we want other providers in the future, stuff like that, we wanna have that flexibility.

[00:10:57]
So what do we do? Do we go into our repo and make a whole bunch of code and then make a whole bunch of other code later? No, what we can do is we can use Terraform's concept of modules, right, and actually reuse those as workspaces. And so the idea is that what you do is you create a network module that's just for Amazon, right?

[00:11:20]
And then you create a workspace with parameters for that network. Now, the idea is that this network, it doesn't have a name, anything that's unique to that network, it should not be there, those should be the configuration parts. And then when we use that module with Terraform Cloud, we can tell it, hey, Terraform Cloud, pass these values down to the module.

[00:11:42]
And so what this effectively now makes it is that in this scenario, going back to the whole repo and two steps. This actually makes it so you only have to make one change to create multiple networks. Does that make sense? Because the idea is you make it once, and then you just reuse this repo over and over and over again for all of your AWS networks, right?

[00:12:09]
Now, here's the fun part. Say I wanna add something to all of my networks, is that hard or easy? What do you guys think? Say I wanna add a tag to all of my networks in Amazon, do you guys think that would be difficult or easy?
>> Easy.

[00:12:27]
>> Easy, why?
>> It's-
>> Yep.
>> One module.
>> Exactly, yeah, that's the cool part about this approach, is if you create five workspaces with five different networks, they all track the same repository. So if you make a change to that one repo, it will propagate and trigger pipelines for all of the workspaces that track that repo.

[00:12:51]
So this is the next part of why this modularity is so nice. Because what you can then do is you can say, okay, well, let's make it so everything in preproduction autoapplies, right? So we can keep making changes, keep testing things, and we don't have to touch things.

[00:13:05]
But in prod, we will make sure that those manual applies so that they go out in their own release cycle with their own changes. And we can handle that better, making it easier to make those changes to production. That concept right there is how you build cloud infrastructure.

[00:13:23]
That is really the sole part of making it so that you can be multi-cloud, right, and making it so that you can create resources very, very fast. And again, you'll see how we do that here shortly. But then because we have this concept now, we can apply it to other things.

[00:13:42]
So we have a network now, so say I have an Amazon network that I created, cool, now I wanna make an Amazon cluster. Well, would I add it to the Amazon network repository? No, because I wanna make more than one cluster potentially in the future. I wanna have a prod cluster, I wanna have a dev cluster, right, things like that.

[00:13:58]
So if it's reused more than once, it should be a module, right? And if it's only used once, it can just be a source repository. But you'll see how I'm using both concepts here, where I'm saying these two, they only manage one thing. So we don't need modularity or any crazy thing there, they can just be that.

[00:14:19]
But these ones, we need to make sure can have names passed to them so that when we create a network, it has a specific network name. When it's a cluster, we can tell which environment that cluster's for, right, things like that. However, and again, because it's all in one repo, when we wanna create a new cluster, we don't actually have to go to this repo to do it.

[00:14:43]
We now can lean on the power of Terraform Cloud and just create a workspace that tracks that repo. And the only difference between those two workspaces are the parameters that are passed to the module underneath it, that's it. So in five minutes, I can create a network with over a thousand resources in it just because I'm reusing a module that I've already made, right?

[00:15:07]
Now, the last one, the product service one, this one is a little interesting because this one actually kind of merges both of them together, right? We said, okay, we've got workspaces for one to one, we've got workspaces for one to many, but what about workspaces for many to many?

[00:15:25]
What about we have a scenario where I have many services that have to go to many cloud providers? What do I do in that scenario, right? Well, and I will say this, it's a bit up to you, there's two different approaches, you can take this, but this is the one I'm taking for us.

[00:15:41]
Which is in the last repo, we're actually creating a repo that will handle service creation on any provider. And inside of that repo, we're going to create reusable modules for each provider. Now, in this scenario, the module isn't here, right, the module is actually in the repository. However, that's not the problem we're trying to solve, we're just trying to deploy our services not to a specific cloud provider.

[00:16:10]
And this is where I say you can kinda diverge here if you want. You could also say, AWS-product service. Can anyone tell me what that would mean?
>> Product is an app, functionality, whatever within the company.
>> But it's on?
>> It's in Terraform.
>> No, it's on?

[00:16:32]
>> You mean that the clopper lighter that you choose?
>> Yes, it's on Amazon, yeah. And so what I can do is then I can say, okay, well, Amazon, Terraform product service has a, I'm sorry, there's a module inside of the product service repo for Amazon, right? And then there's also a module for Google, right?

[00:16:52]
And what we do is then we can say, well, I have an array of services, all of these services use all of these modules. And again, that might not make a lot of sense yet, but the TLDR is that we're creating modules in the repo instead of using a module directly to reproduce things.

[00:17:12]
And the main point is so that if we have a bunch of services we wanna update all at once, this avoids us having to create a bunch of workspaces. Cuz that's another alternative is you can just create a project per service, or yeah, per service, but then it's also per environment, right?

[00:17:31]
Whereas we only really care about environments more than services and environments. And so in this case, the single repo solves the problem of saying, these are all the product services in dev, right? These are all the product services in staging, right? And then this is all their automation inside of it.

[00:17:50]
Questions, does anyone have any questions on what you see in front of you? So the main takeaway here is that these are components, they're reusable. And as long as we follow this methodology of saying what problem are we trying to solve, then we can create components in a way that they solve our problems, right?

[00:18:10]
We said we needed networks, there's our network module, right? We recreate networks with that module. We need clusters, there's our cluster module. We recreate clusters with that module. We need services, cool, let's put all the service automation in one repo, deploy that to an environment. What does that mean?

[00:18:31]
That means that you can take one repo of automation and deploy your entire service layer just based off of the variables you give it, right? So this means that, again, when I'm ready to deploy product service to staging, I just tell the workspace it's staging. That's it, that's all I need to do.