Transcript
We're going to be talking about how we can centralize our Secrets management system for our application teams without Impacting their applications and causing undue risk. Also, how can we do this in a way that we can allow our Application engineering teams to manage that system? Now, before we go too far, some of the things you're going to See today and some of the demos we're going to do have not Formally been released to the general public. So you're the first to see them. So thanks for coming today. Also, at tomorrow morning's keynote, applause and clap like You've seen it for the very first time, please. All right? all right. So let's get started. If you would have asked me at Hashi comp 2023, is it a good idea to centralize your secrets In one system? sure. But there's going to be an impact to your application Teams. They need to rewrite their Applications to point to this system. Your platform engineering teams Need to support this process. Is it worth the squeeze? I'm not sure. Now, hashi comp 2024, i would Still say yes. This is a great idea. But if you'll give me a mic, 30 minutes, and a platform, i'll Show you how we can do it and minimize that risk and eliminate Any undue support by our platform engineering teams. So while we're talking about secrets, we've all got secrets. We've got secrets in our databases, in our api keys, Connection strings. We've got secrets held in a csv File we don't want to talk about. We've got secrets in aws Secrets manager, azure key vault, github variables, and Azure devops. This is the definition of secret Sprawl. What we want to do is onboard Our application teams into one centralized secrets management System so we have a source of truth for our cyber team to Point at. We need to ensure that access on That system is minimized to just the proper access so that Services are only accessing their application secrets and We don't have a sprawled access pattern which we typically begin To see over time. We need to ensure that our Systems and our applications can integrate with the systems Management platform in a way that's not disruptive to their Current day-to-day activity. And lastly, our application Teams still need to manage those secrets. They still need to create, revoke, renew, and retire those Secrets at some point in the future. And we've got to make this easy for them. Before we go too far, i'm jeremy myers. I work at gm financial. I've been in the i.T. industry About 20 years, 10 years using the hashy suite of products Starting with vault and terraform on prem, moving to Hcp terraform and vault, using packer and waypoint as well now. So back to secrets. Let's just clarify secrets. They are sensitive information to our business. They're critical for our applications to use and connect With other services. They're necessary. We can't get rid of them. With workload identity Separation, some of that goes away. But generally speaking, we won't fully eliminate it probably Anytime soon. So these secrets should not be Shared. Application a, application b, if B is compromised, it should not impact our application a due to The credentials being, you know, invalidated. Whatever we do today should be easy. It should be easy for our application teams to onboard. It should be easy for them to manage. And it should be easy for our platform teams to manage long Term as well. So let's get to using hcp Waypoint for our automation to do so. And let's use vault secrets as our source of truth and secrets management. We'll leverage waypoint for the automation. It's build to fit. We'll use terraform no code Modules to do this. And we'll leverage different To roll out our hcp vault secrets. Why vault secrets? It's serverless. We don't need additional head Count to support infrastructure. We don't need to patch it or Version it. We don't need to do anything Other than stand it up and enable it and get our teams in there. It's ready on day one. We have granular iam access Controls, which is being rolled out today. And we're also looking at three different types of secrets, one Of which is new, but you're probably familiar with if you're In the room. This is big. But for those who don't know waypoint, waypoint has three Primary capabilities. Templates. Templates are the core underlying infrastructure that build Applications. Remember that. Templates build applications. This is your core piece of Infrastructure. Next are add-ons. Add-ons enhance an application. And add-ons will be added to an Existing application. And lastly are actions. Actions are custom builds, scripts, api calls, whatever it Might be. We saw arman do some of this This morning. But these operate and maintain This application. Now, for today's purpose, i Primarily use templates and add-ons. I could have leveraged actions. It just didn't fit in the time. So i'll point that out when we get to it. Why hcp vault secret capabilities? well, first off, it handles three Of the most common types of secrets. Static. Think key value pairs. Simple, straightforward. We all know them. Next is auto rotating. You guessed it. 30, 60, 90 days. Auto rotate up to 50 versions And up to two active any given time. This matters. Why? Because an application that's currently running, we cannot Retire the secrets it's actively using. So we create a second version and allow the application to Refresh, reboot, restart, recycle, whatever it might be, And recache those credentials at a time that's convenient for Them, perhaps a change window. Next are the secrets sync. This is the key piece you want to pay attention to. We can sync those secrets from our cloud secrets app down into One-way push, not bidirectional, one-way push down into our Native cloud secrets management systems. Things like azure key vault, secrets manager, gcp, you can Read, you can see it. I left one off. Terraform cloud variables are also in this list. We can push credentials from the vault secrets app into our Terraform cloud credentials, which i actually leveraged Today. I left it off. And lastly are the feature set. It's serverless. Again, we don't want to be patching os anymore. This is not fun. It's not something we enjoy Spending our time on any longer. Multitenant. Secrets apps allows for api calls, cli calls, terraform Calls, data calls or resource calls. You can call it from anywhere. Next is workload identity Federation. We solved the secret zero Problem before it existed here today. And lastly are the refined im access controls. I'll grant your application access to your apps and the next Application will only have access to their apps. There won't be any cross contamination of sharing secrets. So let's put on my platform engineering hat today. I like to build everything as code so it's repeatable and Ensures it's consistent and my peers can deploy it tomorrow as Well or while i'm at hashy comp and they get the same result As if i was in the office. Great. We need this to be on demand as well, by the way. So what does it look like? i'm a platform engineer. I'm going to build a terraform no-code module. That's a requirement for templates. If you're not familiar with No-code modules, it means you have a provider block. We don't typically put provider blocks in our modules or you Shouldn't be. That means we need to provide Credentials for that module. We need to give it a way to Authenticate to whatever system it's calling. I'll push that module up into my git repository. I'll connect it to my http terraform registry and Designate it as a no-code ready module. At that point, i can connect waypoint into my http registry And tell it which no-code module to leverage for that template. Today we'll be generating http projects, secrets apps and Injecting secrets into that app, all using templates. So what does it look like? in our registry, here we are. We've got a no-code ready indicator by a cool rocket ship, Which i love, and i've got three of them. So i've got one for projects, secrets apps, and secrets Themselves. What does the project template Look like? this is simple stuff. You've probably all seen it before. I'll create the http project. I'll generate a service principle that has admin so it Can generate other service principles in the future and Secrets app manager so it can generate secrets apps. Also generate a key. Remember, we're working with a No-code module that needs credentials to be supplied to it. So over in my http terraform cloud, i'll generate a project. I'll generate a variable set and associate that variable set To the project. This template will spin up Workspaces within this project that leverage the credentials Assigned to it. So we don't need to worry About the application team supplying those credentials or Storing them anywhere other than in a secure place like the Terraform variables. That's next. Let's add those credentials in there. We've got client id and client secret. Pretty straightforward. Let's flip over to waypoint. We need to connect waypoint to our http terraform registry. So we do so here. We connect the connector. And we tell it how to build a template for http projects. And if you're looking real close, you'll see that's Actually vault secrets. Wrong one. Sorry. But it's the http project that We're creating here. And at the end of the day, what We end up with is an http project. Now, vault secrets, on the other hand, will spin up our Vault secret app inside the project we just created. We'll then create two service principles. Remember, refined im access controls. Two service principles. One with reader access for our Application teams to use. And one with manager access to Create those secrets and manage them, rotate them, revoke them, Whatever it might be. We'll generate our spn key. And, again, we'll generate another terraform project with Variable sets and connect those. We'll generate the variables Within the variable set with our client id and secret. And, lastly, it's generating secrets within the secrets app. We can use terraform to do this. I'll show you how we do it in The portal as well. But here's terraform to create A static secret as well as a dynamic and auto-rotating secret With aws iam credentials and rolearn. Pretty straightforward stuff. Pretty easy. I don't recommend you click ops this because we like things as They are. Here's how we create a template In the portal if we were going to. We give it a template name. This needs to be url specific because we can trigger it from The api technically. So it needs to be capable to be Handled at the url level. We'll give it a summary name For our application teams to know what this thing does. And we'll choose which terraform module to point it at. Remember, these are no-code modules. And so if they are not designated as a no-code module, It won't appear here. We'll tell it which terraform Module we want the cloud project to leverage, which variables need To be filled out by our application teams, and which Ones we will prefill. We say next. Give it a very lengthy description to fully describe What we're doing here for our application teams. Control v. And then because we're all Good platform engineers and we write good read me's, we'll Import that. And we will give it a tag because We tag everything. We'll say publish. We now have a template that our application teams can leverage. And we can see the resources it will generate. It is ready to go. So let's switch hats. You're the application team now. And you hear that we're Centralizing our secrets. Your requirements and objectives Are make it self-service. Make it easy for me to onboard When i'm ready. I don't want any additional Overhead of submitting tickets or waiting on other platform teams To do the work for me. I want to do it myself. Next, i want to leverage my existing services. If i'm using azure key vault today for my application to get Secrets from, i want to continue to do that. I don't want to change my app code. If i'm using github variables for my pipeline, i want to Continue to do so. I don't want to move the Cheese, if you will. And lastly, i want to integrate My existing processes around my pipelines and make sure they Still have access to the variables they're supposed to. So how do we do it? application team comes in. They use waypoint automation to generate the project and the Vault secrets app and inject secrets. They come around in the afternoon, and they can use Terraform to leverage those secrets or from their app code They can get to those secrets through files or integration Syncs, which we'll look at here in just a bit. Diving into that just a bit more, using waypoint to Generate a project, the output of that will be an hdp project. And as a result, they'll get a service principle that we Looked at that has access to create other service principles And secrets apps. Now, real life, what does it Look like for them? they'll go in and create an Application based on the template that our platform Engineering team deployed. They'll say next. They'll give this an app name. This is a waypoint app name That will appear within the waypoint portal. They give the description and tell it which project within Hcp that they can leverage. This could be a drop-down. You can supply predetermined variables and provide a Drop-down if you wanted to. We'll give it a few seconds to Run here. That's all it takes. This is self-service, and it's easy. We've already built the underlying Infrastructure behind the scenes in terraform, so all it's doing Is generating a workspace, running the terraform plan and Apply for the app team, and spinning up these resources for them. At the end of it, which should be very soon, we have outputs Coming, and what we'll have is an hcp project with a service Principle associated with it. That didn't take too long there. What we end up getting is a project called sales department With service principles inside of it, scoped to our sales Department hcp project. Remember refined iam access Controls here. We have our secrets that we Created over in terraform that exists now for our application Teams to use. Next, they need to create an Hcp vault secrets app, and we'll use a template to do that as well. We'll generate two service principles doing this. So what does that look like? again, they'll use a template, Say you create a vault secret app and give it a waypoint app Name here. We'll tell it which app to Leverage. It will be the sales web ui That we'll create, and we'll give it a description to indicate Which vault secrets app it is for. They'll say create application, and we'll wait just a few Moments, and what we'll have are many resources now within the Hcp project that we generated earlier. We'll have a vault secrets app, two service principles, and Credentials that will allow us to add secrets to that vault Secrets app. There it is. Success. What do we have? We've got our project in the upper left-hand corner for a Sales web ui vault secrets app, and we have two service Principles with the appropriate level of access. Now they need to inject secrets. They can use terraform to do That, as i showed you earlier, or they can use the hp cli. What we're going to see here on the hp cli is that i generate The same secret twice. We'll see the versions begin to Increment here. So version one was created Above, and now we have a latest version two as well. This is great. But let's see how they can do That within the portal. Again, this is an add-on. Remember, an add-on enhances an application. So our existing application was our sales web ui. We'll go to add-ons, and we'll install an add-on. This is our secret. We'll give it a secret name and Value. Fairly straightforward here. These would be your static credentials. These are your static sources of truth here that we need to Enter into the secrets app. There we go. We click install. And we wait patiently for it To finish installing that. And what's occurring in the Background is this module is leveraging the existing Credentials that we built out within our hp project to access That specific vault secrets app to inject those secrets. Now, this doesn't take too long, but once it's done, we'll Rotate screens, and we'll flip over to our vault secrets app And see that we do, in fact, have a new secret that was Generated by this process. So once that's done, we'll flip Over to vault secrets app. We'll do a little refresh of The screen to ensure that we get all of our latest secrets Showing, and we do, in fact, get a secret for our hash account Key. And we can see that secret does Exist, it is accurate, and it is in its place. And that's how our application teams can onboard themselves Into leveraging the new secrets app to get their secrets up to Par and in the centralized system. Now, auto rotating and dynamic secrets. Some of you might want to pay attention here. These are very common scenarios on vault dedicated, but they're New to vault secrets app. So auto rotating here, our First one, we're just going to do a read of our dynamic Credential we have in the background. This is calling aws, using a roll arm to generate my Credentials. For all you ethical hackers, I've already revoked the tokens. There's no need to go and try. Okay. What we'll see here is that we Generated credentials on the fly. We dynamically got them and Generated them. Now, how about rotating? I have a rotating secret now. I'm on version 4, and i'm going To go ahead and trigger the rotation of that. We can see that it successfully triggered the rotation and vault Secrets app. I can have many versions, up to 50. In this case, i got impatient. It's still on latest version 4, So we'll try it again after patiently waiting. And perhaps we'll see version 5. There we go. How about that? now, how do we know that it Actually rotated that? because i have a sync going in The background that needs to sync it to aws secrets manager. Let's take a look at the access key. Hs4m is the latest for version 5. Cool. Let's flip over to aws secrets manager and see if it did, in Fact, copy that. Oh. That's just, never mind. We haven't looked at that yet. So let's talk about file secrets sync, actually. We're going to sync those secrets from our vault secrets App after we rotate them into these different systems. And this is where we're not moving the cheese for our Application teams, right? we're going to ensure that they Can continue to leverage the applications and native services That they have been until today. And terraform variables or Secrets manager or your kubernetes secrets operator as Well, we can inject those secrets into those services so That your application continues to leverage native cloud Services and leveraging the im policies and capabilities within Those clouds. Now, there's a tricky one with Azure devops, if you're familiar with that. We can't inject directly to those, but we can inject into an Azure key vault and your azure devops will connect to a key Vault. We make it there. Now, if your system isn't listed here, that's okay. We have hbcli. We can always make the call, as We saw, to read the secret, rotate the secret, and then Inject it over to whatever third-party system you may be Using. So it still will work. Let's see that in action, as i mentioned earlier. So we'll look back at our rotate, and we'll see that we're On version 6 this time, and we will instantiate the refresh Update of that secret. It fully went. I'm a little more patient, i believe, this time, and i wait Just long enough for it to rotate, and we get a version 7 This time, and we'll see that it is actually an aws secret key Of what? let's look. We'll open it up and see that it ends in 2cv6. It is version 7. Let's flip over to our aws Secrets manager and verify that did, in fact, get synced. There it is. 2cv6. Our secrets are now syncing from azure, not from azure, but from Our secrets app down into our aws secrets manager, where our Applications, lambdas, ec2s, whatever it might be, can Leverage those. So that's a good thing. We haven't forgot about our security team, though. We need to ensure that our security team has visibility Into everything and understands what's happening on our secrets Management platform. They want logs, they want audit Trails for compliance, and most of all, they need to know the Who, what, when, where, and how. We haven't forgot about you. We now have audit logging now on the vault secrets app. We can tell what was going on and we can tell who and when it Happened. Starting at the bottom, we can See that i created the vault secrets app using my service Principle, using terraform. That's pretty cool. Next, i went into the ui because i was lazy, and i created the Vault secrets sync and generated a secret and forced that sync Down into the sales web ui that we created. I then attempted to go from the cli and access that secret. I was successful and attempted to access a second secret, which I didn't have access to, and that failed. We can export all of these logs to splunk, data dog, and cloud Watch, so the end point can be whatever system you are using. So this is good. And lastly, our granular access Controls for our security folks. You need to ensure that we're Only accessing the secrets that we were meant to access. We can see here the vault secrets app manager has project Level, which is denoted by the cool little notepad icon, and it Has admin access and vault secrets app manager. While the sales web ui reader and manager only have access to The vault secrets app that they are designated access to. It's a good, refined iam access controls here. Let's circle back to the beginning here. We said secrets were sensitive, and they're necessary, and we Need to ensure that they're not shared. We leverage vault secrets app to do all those things by ensuring They're secured and separated within our apps. We also said it needed to be easy. We've made this easy using Waypoint automations for our application teams to onboard With. Our platform engineering teams Are already writing terraform, so they can write our no-code Modules for us, and it's easy for them to support, and we've Leveraged a common terraform platform that we're already Using for registry anyways. I hope you learned something Today. I hope that you'll take a look Into these services, and if you have any questions, let me know. I'll hang around in the back. If you want to connect on LinkedIn, there's my qr code. We all need a good paired Programmer, but mine's a little bit more lazy than most. Thanks for coming today.
