Transcript
Mark Tinderholt and welcome to HashiTalks 2024. A little bit about me. My name is Mark Tinderholt. I work for Microsoft, but I'm not speaking in any official capacity today. I am just really passionate about two things, Azure and Terraform. And I try and spend a little bit of my time sharing that passion with others and helping others learn Terraform and how to Terraform Azure. So why am I here talking to you today? I think this picture gives a pretty good reason why I'm so passionate about Terraform. I come from an AppDev background and I can't tell you how empowering it feels to be able to design my own environment and code my own environment so that it is 100% repeatable every single time, no matter what platform I'm targeting. So I created this little meme because I think it's pretty fitting, you know, and it's like what tool can really say that they can do all these things and be just as good at it as anybody else, right? So not that these other tools are bad, but I mean, you're at HashiTalks 2024, so you know, maybe we agree on this anyway. I abide in sunny Columbus, Ohio, where I organize the Azure CBUS meetup and the HashiCorp user group of Columbus. So I have a good bit of fun doing that. If you're ever in the Columbus area, check meetup and see if we have any events going on while you're in town. I also try and share my knowledge online through my YouTube channel called the Azure Terraformer. Most of my content is educational, but I do try and have a little bit of fun on there too. So if you're interested in Azure Terraform or just Terraform and automation and cloud architecture in general, feel free to check me out. But what am I talking to you about today? Today we're going to be looking at Terraforming AKS, or Azure Kubernetes Service. Design considerations and best practices. Of course, I have to emphasize these are my own personal opinions. I don't speak in any official capacity, so take that for what it's worth. The tech stack that we're going to be working with today is, of course, Azure. We got GitHub Actions in play. We have Terraform and Kubernetes, and the application code that I use to deploy stuff is .NET. All this code is available online in a GitHub repo, fully working. You can go check it out, clone it, and work with it while I'm giving my talk or when I'm done. So the high-level architecture that we have, we have a few little microservices that we want to deploy. A user service, a tenant service, a health check aggregator, and we want to deploy them all to our AKS cluster. Now, we are going to be using Key Vault, Azure Monitor, and Cosmos DB. And to front-end our application, we've got Application Gateway and Azure Front Door. So traffic comes in through Azure Front Door, gets routed to our Application Gateway back-end, and hits our AKS cluster, which uses an NGINX ingress controller that routes traffic to the appropriate pods. My talk today is going to assume a certain amount of knowledge about these services and about cloud architecture and about Kubernetes and Terraform in general, because we are going to be focusing on how to do this, how to set this stuff up at a code level and configure AKS according to best practices. The source code that you'll find out in my GitHub repo is pretty much grouped into four buckets. We have GitHub actions workflows, which are a bunch of YAML files in the normal place that they're in. We have two sets of Terraform code, one for our Azure infrastructure, and then another for our Kubernetes configuration, where I'm using the Kubernetes and Helm providers. And then my application code is in a folder called .NET, and you can see all the different projects and solutions in there. Consequently, there is a Docker file in each of the services that we deploy, so that we can push images up to our Azure Container Registry and then deploy them as pods within our Kubernetes cluster. The pipeline structure that you'll find, again, we have two Terraform code bases. One is an Azure code base that we have a plan and apply pipeline that provisions our Azure infrastructure. Then we have this Docker image process where we build and compile our .NET code, push Docker images out to our Azure Container Registry. And then our second Terraform pipeline is going to do plan and apply, but this time we're going to be using Kubernetes and the Helm provider of Terraform in order to actually provision things within our Azure Kubernetes cluster. Now the agenda for today, bear with me, I only have 30 minutes, so there's only so much that I can tackle within that time period. But the most important things that I thought we should talk about today are these six items. There's plenty others out there that we could really get into, and I encourage you to check out my code on GitHub to see all of the different configurations and see actually the whole end-to-end CICD pipeline. But what we're going to focus on today are these six things. First, we want Entra integration, that is both operators using the AKS cluster, using kubectl, as well as our workloads that are deployed onto our cluster in pods, such that they can access other Azure resources out within the environment. The solution that you'll find on GitHub has Key Vault, Cosmos DB, and Azure Container Registry, to name a few. And we want to make sure that both our Kubernetes cluster and the pods themselves can access those things without messy secrets or credentials being embedded within the cluster. The second one is private networking. We want to keep our AKS cluster as secure as possible from a network standpoint, so we're not going to be enabling public endpoints on any of the Azure services that we provision. We want them to be attached to our private network, such that only the AKS cluster and the pods therein can access them. Next, we'll be going through Key Vault integration, which is going to allow the pods within our cluster to access secrets stored within our Key Vault and use them within the .NET code. Next, we're going to look at how to deploy our cluster in a zone-resilient way. There's a couple of different strategies for this, and I'll go into those very briefly. And then there's also a technique where we can apply a new preview Azure policy to actually check our entire solution end-to-end to make sure that we've applied zone resiliency across all of our Azure services. Number five, of course, we have to talk about observability. Of course, there are many, many different ways of making this possible and leveraging different observability stacks. I'm going to showcase just kind of the standard Azure Monitor stack, and we'll talk through how to set that up. Lastly, I want to talk about maintenance plans. This is an often neglected capability of your Azure AKS clusters, setting up both the system Kubernetes services to be auto-upgraded, but also the underlying operating systems that are running on the nodes within your node pools. So we'll take a peek at that and how to configure that through Terraform. So these are really the six focus areas for my talk today, covering six different aspects of best practices for configuring Azure Kubernetes service. So here we go. First, let's start with Entra ID integration. Well, of course, the first thing that we need is an AKS cluster, and we can do that with the Kubernetes cluster resource within the Azure RM provider. I won't go into all the details of the resources that I provision. You can go check out my GitHub repo for all the code. It's fully working. You can see all the things. So during my talk, I'm really just going to focus on the key hotspots within the code. Next, we need to set up a managed identity. This managed identity is going to be used by our cluster to go provision all the underlying infrastructure that Kubernetes needs to operate on. So a lot of networking things and stuff like that. Then we need to assign that managed identity to our cluster. We can use this identity block to specify the user assigned managed identity and to point it at the one that we just provisioned. Now that we have the AKS cluster set up and the managed identity provisioned, now we need to create role assignments to grant this managed identity access to the things that it needs to do, and that is a network contributor within the Azure virtual network that our AKS cluster is provisioned. Another thing that we have to do is we have to grant the kubelet identity that is automatically provisioned when we provision our AKS cluster, ACR or Azure Container Registry pull access. This is going to allow that internal service of Kubernetes, the kubelet service, to be able to pull images from our Azure Container Registry. So we can do that by creating our Azure Container Registry, but then also granting this ACR pull role assignment to the kubelet identity that we pull off the outputs of our Kubernetes cluster. So far, we've been connecting with EntraID, but not explicitly. We've been using Azure Managed Identity to do most of the work for us. Now we really want to be able to enable role-based access control and EntraID entities to be used within our cluster. And in order to do that, we need to enable role-based access control and add this active directory, EntraID role-based access control block, in order to create this capability within the cluster. We can also specify some administrative groups to grant ourselves access to the cluster. So if we have certain sysadmins or operators that are going to be needing to access the cluster and log in and use kubectl and things like that, this would be a great way to do that. You can, of course, check out my code on GitHub and see how to do that for yourself. But basically, you're going to be using an EntraID object ID for an EntraID group that's out there in your EntraID directory. So now that the cluster operators have access through EntraID, now let's enable our workloads to authenticate using EntraID. So to enable workload identity, again, we're going to be setting some attributes on our Azure RM resource to enable both workload identity and the OpenID connect issuer. Next, we need to set up a new user assigned identity to represent our workloads and then create a federated identity credential that Kubernetes internally is actually going to use to connect through to this managed identity. So the federated identity credential is actually what's going to be tying the Kubernetes layer to the Azure layer. And this is a critical step in the process because the user assigned identity is actually what's being granted access to the various Azure resources, but this federated identity credential is actually what's connecting Azure to the AKS cluster, to the Kubernetes resources within the cluster. And as a result, we need something within Kubernetes to be provisioned that's going to reach out and touch that federated identity credential and indirectly that managed identity. And that is a Kubernetes service account. Now you'll notice this is where I'm going to be using the Kubernetes provider of Terraform to be provisioning this resource. So this is after our cluster has been provisioned. We connect to the Kubernetes cluster, we create this service account, and we use these special annotations in order to reference that workload managed ID. Now when we want to use that managed identity within our pods, all we have to do is in our pod specifications, add some additional labels like, hey, you got to use workload identity. And by the way, use this service account, which is already connected to our federated identity credential, so that we can use this managed ID. And there we go. Our pods are now able to impersonate this managed identity whenever they talk to different Azure resources. And we can just add role assignments to grant access to different services that we provision within Azure, whether it's Event Grid, Cosmos DB, Key Vault, etc. And that's it for intra-ID integration. Now let's talk private networking. So where to begin? Well, we have an AKS cluster, we have a virtual network that that AKS cluster sits on. How do we connect our Azure services like Key Vault or whatever to this virtual network so we don't cross the internet in order to access it? Something has to be in the middle to connect these two things together. And that's what a private endpoint is all about. Now private endpoints can be a little bit tricky, so I wanted to break this down in great detail. So let's remember we have our Key Vault and we have a virtual network and we want to connect to it. The first thing that you need is a private DNS zone. Now this private DNS zone is going to be different for each service, so you're going to have to look up what that private DNS zone should be for each Azure service that you want to do this for. In this case, private-link.vaultcore.azure.net. After we set up the private DNS zone, we need to link that private DNS zone to a virtual network. And this is a very simple resource. It just takes your private DNS zone and connects it to your virtual network. Now that we have that set up, we're ready to actually get into our private endpoint. And the private endpoint is kind of that man in the middle, so to speak, that connects the private DNS zone to the subnet to the resource. So it's truly the crossroads of this private network connectivity. And you can see on the private endpoint resource, we have those three connections. We have a pointer to the subnet. We have a pointer to the actual resource, which is going to be different depending on the type of resource type that you're using. In this case, we're using Key Vault. And then we have a pointer to our private DNS zone. And that's it. This will essentially set up a network interface card that's on this subnet and will connect this Key Vault on that private IP address within that subnet. The DNS resolution will automatically handle, so you can continue to use whatever the public Key Vault endpoint is, and it will get routed to the appropriate place as long as that traffic is on that virtual network. And you can check this out in the code base that I shared through my GitHub. I set up private endpoints for several other services within that same solution. And if you're looking for other services, pretty much you just need to look at what the private DNS is and then look at the different sub-resource names. Each is going to have different sub-resources depending on the resource type that you're looking at. Key Vault has Vault, and other services have other sub-resources. So, definitely go check those out. So, we've set up private networking for our Key Vault. Now, let's actually get Key Vault integration going within the Kubernetes cluster. So, the first thing that we have to do is we actually have to enable Key Vault within the AKS cluster. This is an extension block that we can add to our AKS cluster resource. And all this does is just turn on the ability to connect it to a Key Vault. This isn't actually pointing at a specific Key Vault. That comes later. And that requires something to be actually deployed to the Kubernetes environment. But without this block enabled within your AKS cluster, you're not going to get very far. Now, the next thing you have to do is you have to provision a secret provider class. And the secret provider class is a special type of Kubernetes resource that will actually connect your Kubernetes environment to a specific Key Vault. And you can see in this manifest that there's a couple of things that are of interest here. Number one, the API version is a little bit different than other standard Kubernetes resources. So, pay close in mind. And that's another reason why I have to use the Kubernetes manifest resource within Terraform and not some other more well-known resource type within Kubernetes. The next thing you'll want to pay attention to is the client ID and the Key Vault name. These are, of course, pointers within Azure to Azure-specific resources, managed identity that we're using, and the Key Vault that we actually want to talk to. And then, you also have to explicitly specify which secrets within that Key Vault that you want to use. And you can see I'm just using AppInsights connection string right here. But if you've got more of them, you're going to have to add, you're going to have to enumerate all of the secrets that you want to reference. And you can go check that out in the actual code base on GitHub. Now that we've got our secret provider class provisioned to Kubernetes, we simply need to reference it from a pod. And so, in this case, we need to update our pod specification to add a mount that references that secret provider class. And it's actually going to be mounting a volume onto our pod that has those secrets embedded within it. But this is largely transparent because essentially those secrets are going to be exposed as environment variables that we can pull in within our .NET code or whatever code that you're using. In my GitHub repo, you can look at how my .NET code is essentially using .NET's configuration libraries in order to pull in those environment variables that the secret provider class sets up for us. And that's it. And now my pods have access to my Azure Key Vault through that Azure managed identity. No secrets, no certificates to maintain. All managed by Entra ID and the Azure platform. The next thing I wanted to talk about is availability zone resiliency. This is something to consider in general when designing for cloud. You should always plan for failure and design for failure so that your systems and applications can overcome failure. Now Azure provides you ways to do that, but you have to participate. You have to you have to get involved. It's really not that hard. So Kubernetes operates in some ways on a logical layer and a physical layer. And the physical layer is where it's touching the Azure platform. And largely the physical layer are the Kubernetes nodes or node pools as they're organized. And so within AKS, you can provision one or more node pools. And as you can see, you can point it at different networks. You can configure different aspects of what operating system you want to use at the node level and what size you want those nodes to be. So you can take advantage of different memory, GPU and CPU configurations. You can also configure scalability differently within those node pools. But the important thing from a zone resiliency standpoint is this zones attribute. And here we're basically telling the Azure platform, I want this node pool to be provisioned across three zones, one, two and three. You can specify two, but why not go for three? I mean, with Kubernetes, you're typically going to start with a three node cluster or more. So it's definitely a good idea to stick with three zones. Other Azure services have other attributes within their respective Azure resources within Terraform to enable zone resiliency. A lot of the VM based workloads have this zones attribute that takes an array of integers, one, two and three. Some have flags to enable zone redundancy. And sometimes the skew can dictate whether you can take advantage of zone redundancy or not. So definitely look at all the services that you're using on Azure and figure out how you can enable zone resiliency on each of those services. Because as we all know, we're only as good as our weakest link. So if we take the time and configure AKS to be zone resilient and we neglect SQL DB or Redis or whatever we happen to be using, then if there happens to be a zone outage or something like that, our AKS cluster might be up and running. But if our pods can't talk to our Cosmos DB account, we're going to have impact within our workload. So it's very important to consider that in all the Azure services that you provision. Another option, if you are very particular about having dedicated nodes in every zone all the time, 100% of the time, this is an important thing to consider. When you just attach the zones attribute to an individual node pool, AKS is going to do a best effort on that node pool to place those nodes evenly across the different availability zones. If you really want to make sure that you're on every single availability zone, you need to follow this practice. It's documented. It's in the documentation, but you might have to look a little bit hard for it where you have to set up single zone node pools for each zone and then distribute your pods and your Kubernetes workloads across those node pools. And so each node pool is going to look very similar to each other, but they're going to be targeting a single zone. This will guarantee you presence all the time in each of those availability zones. Lastly, on the zone resiliency front, it might sound pretty daunting, all the work to go figure out, you know, whether you've configured all these Azure resources or Azure services in a zone resilient way. That's why we have this thing called Azure Policy. It's one of Azure's superpowers. I talk about it a lot. It's extremely easy to attach an Azure policy to your resource group or your subscription or your management groups or wherever in order to detect if your Azure resources have been provisioned in a zone resilient way. This is the magic GUID. Check this out. I'm sure I can provide a link in the chat or something like that if you're interested in trying this policy initiative out. It is in public preview. It doesn't scan all Azure services, but it does do a lot. We are continuing to build this Azure policy, so definitely go check it out. Feedback's appreciated. But you apply this policy on your resource group, and it'll immediately tell you what Azure services have been provisioned in a zone resilient way with this wonderful Azure policy dashboard here. Next, I want to talk about observability, and as I mentioned, we're going to be using Azure Monitor, so there's just a couple of flags that we've got to set on the Kubernetes cluster itself to enable Azure Monitor, the OMS agent, Microsoft Defender, and to connect it to our log analytics workspace. Then it's a matter of applying Azure Monitor diagnostics to our Kubernetes cluster to basically suck all those logs out of the cluster and put them into our log analytics workspace. This should be pretty familiar to you if you're doing anything with Azure Monitor or Azure Monitor diagnostic settings. This is the same diagnostic setting that pretty much applies uniformly across all Azure resources, all Azure services. The only difference is the different logs that you're going to set. You can see here I'm using one of my own modules that kind of simplifies setting up an Azure Monitor diagnostic setting, which allows you to basically, in a more concise fashion, pass in a collection of logs that you want to apply to this particular diagnostic setting. Lastly, I want to talk about maintenance plans. Kubernetes has a lot of stuff going on, and in particular there's system components, there's nodes that your workloads run on, and AKS does a lot of that work for you. But there is a feature that you really probably should be enabling or at least considering to enable, which turns on automatic updates to both the Kubernetes system components and the nodes' operating systems. Because our containers, while we can do whatever we want inside those containers, they are running on nodes with their own operating systems, and those nodes are essentially virtual machines running on Azure. So those nodes have an operating system, and those nodes need to be patched as well. So you can set up different frequency, different intervals to hit different days of the week based on your availability. You can control how many of these nodes should be taken off at a particular time. There's a lot of tweaks and different configurations that you can set for this, but it's definitely one of those areas of your AKS cluster configuration that might come up as an afterthought, and you might get smacked with security compliance after the fact. All this code is available on my GitHub, which I definitely encourage you to check it out. As I mentioned, I've got the source code set up here where we've got the Terraform code in an infra project, which has all the things, and then we have our Kubernetes Terraform, which provisions the Kubernetes control plane things, everything from pods to the Kubernetes service account that's used for workload identity, to the key vault secret provider class, to the ingress controller, and then just a few services that we deploy to run a simple application. So definitely go check this out. This implements all of the things that I talked about today in my talk, both within the Terraform code base and in the Kubernetes code base. Also, as a bonus, I included some GitHub Actions pipelines that run Terraform plan, Terraform apply for both the infrastructure that hits the Azure control plane, and then also the application stuff that sits on the Kubernetes control plane. So go check around with that. I've been kind of using that to deploy stuff. So you can see my deployment history. Whenever I make tweaks or changes to the code base, I can just run Terraform plan and then run Terraform apply. And then once that's done, I've set up the Kubernetes plan and apply as kind of separate pipelines that get executed. Now, of course, I'm using manual triggers within my GitHub Actions, which allows me to basically manually control when I execute a particular pipeline and what environments I target. But if I were working on a team, I'd probably set this up with some Git flow around it such that the pull requests triggers a plan on the correct environments and a merge into a well-known branch like main or develop is going to trigger the apply pipelines in a corresponding fashion. But for my lab environment and just to kind of kick the stuff around, and also for you to kick the stuff around, it's a good way to have more control over when things get executed. So definitely check it out and would love to hear your feedback if there's anything I'm missing as well. And I was always looking to learn more. Anyways, I think I'm at about time. That's it for me. I hope you enjoyed my talk and really appreciate being here. Thank you to HashiCorp for the opportunity to share with you guys how I've learned Terraform AKS. If you want to learn more or hear more from me about Azure, Terraform, cloud architecture, software development, check out my YouTube channel. It's called the Azure Terraformer. I talk about a lot of different stuff, but usually Azure and Terraform are kind of the centerpiece of things. Anyways, I appreciate your time. Thank you so much.
