Transcript
What a keynote. So many exciting new features to try out. Well, not right now. First, Kareem and I are going to take you on a journey into the building blocks of infrastructure lifecycle management. My name is Jenna Goldstretch. I'm a software engineer on the Packer team. I work mostly on HCP Packer and the Packer Azure plug-in. You do a lot of cool stuff. My name is Kareem Satyarli. I'm a senior developer advocate at HashiCorp, obviously, where I focus on infrastructure and orchestration tooling, so Terraform, Packer, Nomad, all the fun stuff, really. And so we want to talk a little bit about history. So let's talk about the last decade-ish. We worked with a lot of customers, and we saw... We've seen a model emerge that kind of works for us, works for our customers. And in that model, we have three distinctive stages, from just starting out to running in a multi-cloud, multi-continent setup, maybe even on edge, maybe your own data centers, many data centers everywhere. And so we call stage one of this adopt. And to better understand it, we have to go back very far, go back in time. And in the beginning, there was nothing, which you can see on the screen right now. And then came the clouds, AWS, Azure, Google, all the other clouds. And quickly thereafter, there was chaos. But 10 years ago, we got Terraform, we got Packer. Right after Nomad came, and chaos turned into codification. And that was beautiful. Codification turned into collaboration. No longer another team's problem, it was a shared problem, and collaboration turned into innovation. So we encapsulated modules, we built code that we could share, packaged up complete applications in container images. And so that collaboration led to containerization. And quickly, containerization led to compliance and compliance issues. We embraced old security models, came up with new ones, and then compliance led to certification. With that, we got a soup of new acronyms. It was absolutely amazing. ISO, CIS, PCI, the European NIS, NIS 2 since October. And you spend so much time dealing with all of this on the fly that you basically have absolutely zero time to watch any of the reruns of CSI. And that really makes me sad. Yeah. We also don't get to enjoy Dora the Explorer anymore because we now have a resilience framework called Dora. So all the fun gets taken away. But compliance is important, certification is important. And all of this was good because, well, when we put more thought into our infrastructure, and into our whole infrastructure life cycle, really, it pays off in a big time. We deployed applications that were better monitored, sought to understand them, sought to understand their character, which we all know our software is definitely personal, it's artisanally crafted, and we love that, right? And so as we adopted these workflows, we were able to do a little bit more. We, again, encapsulated things in workflows called modules. And so as we passed through those modules, we got a little bit towards standardization. And so that very groundwork of thinking about this from the get-go and making sure we stick with the pattern as we build it gave us standardized components. And things were great. I mean, who doesn't love a standard that they built, obviously, because it's going to be the best standard, and something that you can roll out. And all of that was good. It was good. We looked at our infrastructure, everything was green. I loved it. I mean, show of hands, if you remember the time of wallboards where everything was green, Jenkins reporting no errors. Is it that long ago? It's hard to see with the lights, but I'm going to guess that some people still remember that fun feeling, the warm glow of the screens. And so we successfully adopted lots and lots of new technologies, and more importantly, workflows that helped our teams deliver, because ultimately it's never a technology problem, right? People just want to know that whatever we build works, delivers value for the company, so we can sell more, connect more, do something that gets people interested in what our company is doing. But as part of that, we also created tools and processes that put the ownership back to many teams. If you've ever written Terraform, it's something you can pick up in two hours. Doesn't mean that you're going to be writing expert level, and if you're not at that level, you should definitely check out the afternoon session to learn more about proper Terraform module design, but you have the power to build all that infrastructure. And so we went from adopting to standardizing. Could lean back, could lean back and look, gaze upon our infrastructure and know that everything that was deployed was codified, it was consistent, it was compliant, and really these steps forced us to create, sorry, to craft, not just create, we're artists, right? Infrastructure engineers are artists, 100%. Craft better building blocks. And these enabled us to build out platforms and platform teams. There's nothing more fun than going to LinkedIn and changing from DevOps engineer to platform engineer. Even though you're doing the same work, you're writing out best practices, making sure everyone is just benefiting from that. And now with Terraform stacks, it gets even easier, you can have your platform running everywhere in bare minutes. Your finance department might have thoughts about that, but it's a different problem. You're just serving new customers. And so again, all was great. Called it a platform, we shipped it, sounded great at conferences, and we went forward with that. Our infrastructure got deployed, we deployed new apps, our app count skyrocketed, popped up new workspaces everywhere, new regions, new cloud providers, we turned single region into multi-region, and single cloud for many companies became a requirement that didn't satisfy the multi-cloud needs and multi-cloud deployments. But because everything was standardized, it was great. I love that. You know, like that phase of, well, we deployed everything in Azure, works, we deployed everything in AWS, works, thank you for sponsoring, both of you, very important. And it was great until that moment where you see three little dots start animating. And we've protected the identity of the innocence, but your team's chat tool starts animating right next to your CISO's name. And so what I really want to be talking about with you is standardizing a response to what feels like could be shaping up to be a security incident, which we're absolutely trying to catch way before it happens, way before it becomes something that ends up in the news. But also, I want to talk about, we want to talk about how this connects to your infrastructure lifecycle. So as we check out our monitoring tool, our workspaces, all look, most of them look great. Well, one of them, of course, a little bit struggling. So we'll look into that. Gap Closer focuses on some ML cleanups. We want to be working on stuff that responds to questions and to problems with higher accuracy. So we have tools that work on that. And so if we switch to HTTP Terraform, we can get an overview of our workspaces, and you can immediately see that we take this serious. We use our own tools, obviously. It works. And in our overview, we see a lot of workspaces that are well-behaved. Most of these look good. No drift. Health checks are OK. Names not super standardized will work on that. But we're happy. We're happy except for one workspace. Gap Closer not meeting the health check in the way we need it to. So let's have a look there. And with 50, 100 workspaces, it's sheer luck if you're able to find it that quickly. I mean, you could sort on a column and remember that you can sort on a column, but there's easier ways. We already know that we were looking for the Gap Closer workspace. But what if you don't know that? What if you don't know what the workspace is called? What if, for whatever reason, that got cut off in a Slack message you got, somebody copy-pasted something the wrong way? I mean, there must be easier ways. One of those, Saved Views. Absolutely love being able to just create lots and lots of Saved Views where I can say, give me all the workspaces that are on an outdated Terraform version. Give me workspaces that deployed with an outdated Nomad version. 1.9 has been out for 15 minutes. I don't understand why my teams haven't updated yet. Stuff like that is critical. Security is hard. Patching is hard. And there's nothing more joyful than being able to see workspaces like that where I can say, well, I want to see all the non-compliant workspaces or workspaces using old modules. And so if we look here in the non-compliant workspaces, we can see Gap Closer listed as expected. For this purpose, we only broke one. We didn't want to break our whole demo platform. But of course, this view will happily show you all the workspaces that are broken. We can filter on health checks. We can filter on many, many other attributes. And here's the best part. I'm here giving a talk. So I'm going to just select a workspace, create a change request, and make this a little bit more interactive. The way we're going to do this is as we pop up the change request window, we can request our team to take a look. This is true power. This is platform team engineering. I'm going to tell them what to do. It's great. I'm here. I'm regaling you with hopefully funny jokes and interesting stories. And so I don't really have the time to fix this. So as we look into this feature, I'm just going to type in a couple of messages. And we'll start with something like, can you fix this workspace? It's broken. I don't really know what's going on. We have more qualified people to look at that. And so while we know it's a vulnerability, but where, why, how? I don't know. We'll find out. And hello from HashiConf, right? I think that's important. Scroll down, submit, and lower left corner, hopefully, there we go. Your request has been received. The machines are now creating a notification that goes to a team member and communicates to them that there's an infrastructure lifecycle event. Visual deprecation, provider updates, Nomad may be outdated. Any of those things could be happening. And so Jenna gets a notification. Here you go. Thanks, Karim. It's not like I'm also busy giving a talk, but at least this time there's no upsettingly loud pager notification. Now that this issue has been reported, my first step is to check my HCP packer registry to help me quickly identify what went wrong. Here I have my gap closer bucket. I'll click into the version used. Currently, it's assigned to the production channel. On this page, I can see the version details page in HCP packer, and I can unfurl the build card to get more information about my artifact. When packer is configured to track artifacts in an HCP packer registry, it will send the registry rich build metadata about the environment that created this build. We can see the information from the running environment since it occurred in the GitHub action. This can help me quickly track down the build in my CI logs. I have the PR URL. I can easily hop to it. You sure that's correct? Because it says that I broke it. Oh, well, I'm used to fixing other people's mistakes. We can also see about the current Git commit from the directory where this image was built from. And we can also see the version of packer and the packer Docker plugin that was used to create this build. This information grants HCP packer built artifacts level 1 Salsa compliance by providing a basic level of source code identification. Wait, wait, wait. I think this is broken because you're using packer 1.12 for beta. I can tell. It's not my code. So if we switch to our editor interface and pull up the Terraform template, we can see that the nomad job pulls a job spec and renders it using an HCP packer artifact data source and a template file. So let's switch to our packer template that builds the application. We can see we have a Node.js application here, and it uses packer to wrap that application in a container image. Everything looks good there, but I'm not sure what's going on. Maybe I need to debug this more, look at my node vulnerabilities, but maybe not live. So let's not turn this into a live debugging session. We'll just spoil it. Here's a vulnerability I can find in my package JSON file. I run npm update, and the package is now up to date. My security tooling reports that there's no more vulnerabilities. I'm going to quickly make a PR, and I'll make the title fix vulnerability in scraper application, and I'll type out a quick description that explains what I'm doing and why. Let's assign Karim for review. I hope he doesn't mind doing a pull request review in the middle of a conference talk. Yeah, looks good to me. Let's ship it. Let's never talk about this again. Awesome. Let's ship it. Once the new build is finished, after the PR is merged, I can hop back on to HCP packer. Now that I have that new version, I need to use HCP packer revocation to make sure that no new infrastructure is created with the old outdated and problematic artifact. If you recall, this image was assigned to the production channel. Using channel rollback, I can have my prod channel fall back to its last valid assigned version, whatever version it was last assigned to that isn't revoked. Now that I've enabled that, I can revoke, and the image is revoked. I'll go to the production channel and assign my newest image, and we can consume that channel in HCP Terraform. Checking our Terraform config, I can see here that all future deployments will consume the valid artifact from that production channel. That kind of means your job is done. Can we not talk about the package.json thing anymore? Okay. I could have sworn I committed everything. I swear I ran like npm update everything. It worked on my machine, 100%. But so we're back to being standard compliant. Our security tooling reports, everything is good. We'll squash the git commits so it doesn't reveal. It's a team issue, right? We all should care about the platform working. It doesn't matter who broke it. We've finished the groundwork for developing a pattern that allows us to build standardized components, and that's important. Because that's really the path to scaling. And not picking on who broke it. That's important. And so when organizations get to this point of green glowing wall boards like the 1980s, it's usually the point of maturity where they can scale. Doesn't matter if it's multi-continent, multi-cloud, lots and lots of small suitcases with HAI hardware, you want it to be working all the time. And so for that, I'll briefly switch into a different role and show you how we can ensure that everyone uses a fixed version of the code that Jenna just updated. We'll switch back to HTTP Terraform, and in this case, I'm in the private module registry. We've got a module for our gap closer scrapers, which is basically Node.js container images that get rolled out across our infrastructure. In this case, this is an AWS-flavored version, because we want to make sure everything works. And when you prompt the AI that you get useful responses and not where to take your kitten when you're asking about Terraform, you can see that one of the modules in there is our scraper job that Jenna just fixed. Apparently, whoever committed the mistake had only done so two days ago, which is a reminder to not write code on Sundays, because Boston is beautiful in fall. You should be enjoying the beautiful fall foliage instead of working on stuff like this. Walk the freedom trail. That's beautiful. Go ghost hunting. Yeah. And so if we click into that, I might send you another pull request, because there's no documentation here. Okay. Good. Or not a pull request, a change request. I think that's de-prioritized. Right. Well, so we have a non-documented gap closer scraper module. It's great. Not documented, but no shade there. We don't need to talk about it if we don't need to talk about the other stuff. So let's go in and manage this module for us. We could unpublish this module. It would be a lot of fun. Well, pretty much all the teams using this, no matter if it's on AWS or Azure or on the Jetson or in physical hardware that we have running this, would then be enabled to do some chaos engineering. They might not have budget at that time, so we're not going to unpublish today. We're just going to deprecate and make sure it goes a little smoother for everyone. So we're just going to deprecate version 0.2.1, put in a nice message about fixing a downstream vulnerability. If you've ever done an NPM install, you know you're basically pulling in about six gigs of JavaScript files. And we'll leave a note to our internal wiki where we have some more information. Reason for deprecation is really just a summary. We fixed something small. We could put the exact name in there. It doesn't always matter. Here's the link. Go there. Read it there. Much more interesting, right? Hit save. Rest of your team now knows 0.2.1 is deprecated, shows up in the next Terraform run, shows up in the next HTTP Terraform run, and that's important. Every other team can now go and fix this on their own schedule if that's what we want to do. Now, this is a security vulnerability, so we'll probably send a notification on our team channel. We're not going to walk through that flow again because it's pretty much the same. But there's options. We want to be able to say, hey, you know what? This is critical enough that you need to update this right now. But not only are we creating that message, we're also making sure it's very visible. Anyone who goes into version 0.2.1 for this module, for the gap closer, will see that yellow box. You'll see it in the dropdowns. You'll see it in Terraform. There's no escaping this and saying, hey, you know what? I didn't know. And that's good because the quicker we're exposed to old code actually now being considered old and deprecated, the quicker we can fix this. And that really brings us almost to the end of our session. So instead of leaving you with just a link or anything, we want to leave you with three thoughts. In the adopt phase, we brought in new technology, developed new workflows. We developed workflows that are foundational workflows, workflows that enable us to get a better understanding of our infrastructure. And in the standardized phase, we created consistent deployments. We ensured compliance. We got that little pay bump because we are now talking about security. And some of us cross-trained from DevOps engineer to security engineer. Others went the platform route. No disrespect to either. They're both important. But we leveled up. We leveled up heavily. And so we made it to the scale phase. We created work that enabled others to scale, that enabled our company to scale. We built trustworthy self-service approaches. And we expanded on the building blocks of the previous phases. But really, underpinning all of this is a strong culture and an organization that has to remain curious to deploy better, to deploy more securely. And sometimes that's hard, right? When you're dealing with security changes, when you're dealing with requirements that seem never-ending, it gets really, really hard to be, as an engineer, to remain curious. But when you push yourself to go that far, that's where you end up creating all those beautiful green wallboards. That's important. The building blocks of infrastructure lifecycle management are crucial to how you design and lay out your deployments, your organization's deployments. To do cloud right, you can't just get the right tools in. You have to think about this from a unified perspective. Talk to our folks. We've seen a lot over the past decade. Or if you don't want to talk to people because you're queuing for the good coffee, totally worth it. I respect that. Get my colleague Mitchell Ross's white paper, which talks about ILM and much more. Head on over to hashi.co slash ILM with HCP. Get the PDF. Read it. Highlight it. There's so much good stuff in there that it will inspire you to think about your infrastructure better. Awesome. Thanks, Karim. Finally, you may have noticed that our talks are all connected to each other. We highly recommend checking out what Mishra has to share on building an internal development platform that empowers your platform teams. That session will take place right here at noon. And also, if you want to learn how we restored the Gap Closer workspace and learn more about Nomad's GPU plugin and how it can help you with AI and ML workloads, I invite you to join Karim's session in the West Prefunction room this afternoon at 4.15. With that, that's all we have for today. Thank you all for your time. If you have any questions, we'll be happy to answer them outside in just a few minutes.
