Transcript
Today, I'm with my partner in crime. I'm going to introduce ourself first. My name is Welly. I'm a Principal Partner Solution Architect here at AWS. And, Manu, you want to introduce yourself? Hi, I'm Manu Sundarsekar. I'm a DevOps Consultant with AWS. All right. Thanks for joining with us today, Manu. You know what, Manu? I've been talking about AWS CC on HashiTalk for about four times now. It's impressive that Hashi team still allow me to come by today. But I think because I'm bringing you as a special guest in here, we are talking about something a little bit different. So, do you mind to kind of share a little bit more, like a sneak peek of what we're going to talk about today? Sure. AWS CC or AWS Cloud Control Provider, as you said, was introduced with the intention of making sure it's easier to adopt the latest features of AWS. But the underlying documentation was still a manual effort from our side to add, to contribute. So, we were looking at, currently, we will look at a way we have approached to enhance the documentation using GenAI. Awesome. All right. That's a good sneak peek. So, stay in tune as we talk about this more. First of all, I guess we have to talk about why we're working on this, the reason behind it. Maybe for certain folks, the first question you have is that, wait, we have two providers? And the answer is, yes, we do have two AWS providers. It's called the AWS Standard Provider. And what's the new one? We typically call it AWS Cloud Control or AWS CC. So, why do we have two providers? This is the standard AWS provider. You're probably very familiar with this process. You write your Terraform configurations using Terraform core binary. And then it's used the AWS Terraform provider plugins. And then it connects to different types of AWS SDK, whether that's IAM, EC2, S3. If you have a new future services that we launch, for example, then the developer or the contributor will then start to include the provider schemas and the handler to handle those future SDK. Now, this is, of course, a little bit different than AWS CC. Manu, can you maybe explain a little bit about how AWS CC provider works? Absolutely. So, for the first two parts, it remains the same. The configuration that you write is an HTL. You still work with the Terraform binary. The abstraction comes from the phase of when you interact with AWS. In this case, AWS Cloud Control provider is based on Cloud Control API, which abstracts away the service-level interactions. Which means you're not calling those run instance, make bucket, service-specific APIs to create those provision doors or manage the lifecycle of the resources. In turn, you have a consistent kernel API, which Cloud Control API provides you. And on top of it, it absorbs those schema updates and changes so we can build the provider on a weekly basis and get you those new resources and features readily available on the day of launch or close to the day of launch. Awesome. Awesome. Yes, that's a very condensed interpretation of AWS CC. A high level, I guess, is like we have standard schemas available, already published in AWS. And also, along with that, the standardized API is called Cloud Control API, which allows the HashiCorp team to weekly scan the resources and implement the same handler through the API itself, making it almost like a fully automated. Yep. So I guess this is how it looks like, right? So this is the AWS CC Terraform registry. If you are watching this right now, I highly encourage you just to kind of Google AWS CC. Likely the first link is going to get you to the registry page. And if you click around and randomly in some resources, you might see like resources like this. So there's a problem here. And one of that problem is that there's no practical example. This is just the resources, right? Some of the schemas and the attributes that you can use. And then we actually started to add these contributions as well, right? So as you can see in here, it's some of those resources. If you click around and you look at the older version versus the new versions, you start to see this example start to popping up. So the question is, how do we actually edit these examples? I think Manu kind of mentioned this earlier, right? Manu, you've done a lot of this. I think you are one of the top contributor in terms of the examples. Do you want to give a quick sneak peek of how this process works? Sure. So the first thing is start with the resource that you're familiar with. If not, spend time on understanding it. So that's the most amount of time that you want to spend on. Say, let's say, if I'm creating an S3 bucket, I'm probably familiar with the knobs and tweaks that I can add. When I create an example, which is essentially a working example of a Terraform configuration, then I run through the lifecycle of it so that I can create the resource, then delete it, update it, and make sure that provider holds the item potency. Now, that goes to open a pull request against the provider, example-based. Now, the effort that we took is around the resources that you're not familiar with, where you may have new resources that are coming across every week as new schema changes. So you have to keep on doing this across resources that you're familiar with or learn about those. And the effort that is required from our side is that, because you have to dive deep into that resource, make sure that you are creating workable, tested examples, and opening a PR against that lifecycle. Yeah, and it could take a lot of time. Again, as someone who says, if you are doing it for something that you're familiar with, it's easier, but sometimes you don't have the same expertise. So the question is like, we have automation to scan the registry and generate the provider. We have some folks that's working on writing the example, but we're actually not there yet in terms of solving these problems. High level. I think when we started some projects around 2023, we have like 700 resources in the AWS CC. We made very good progress. We added 200 examples since then. But then at the same time, new AWS services comes, new resources is now being implemented in Cloud Control API. It also means it's now available in AWS CC. So we have more resources coming than the example that we added. So this is really what's striking us to start thinking about how we can do differently in these regards. So the question started into like, yep, we have the problems. How can we automate it? And can really generative AI actually help us? And this is actually not our first attempt. Our first hypothesis, this is early 2024, is that LLM start to get better and better. So what if we give LLM the Terraform resource schema in AWS CC and also some sample documentation? So whether that is a similar example in CloudFormations or maybe either just high level documentation about these resources. And ideally, combining these two, giving some prompt, prompt engineering, and et cetera, we hope that the LLM of choice will be able to generate the accurate results. And we can say our first hypothesis was okay, but it's not great. At the time, I think, and still today, like a lot of this LLM has a very long contact length, 200k, right? Which is great, but often like you see the LLM lost the attentions. And meaning that that is that it's very focused on one certain part of the resources and forgotten about the others. And that could cause like some hallucination as well. And that hallucination also occur because AWS CC is kind of pretty new, right? So not many people write a blog post about it. Documentation is there, but it's not as much of the other types of standard provider in terms of blog posts and et cetera. And this initial hypothesis and test, you can check it out in this GitHub repo is basically Jupyter notebooks. Still require a lot of human interactions, basically to come back and then make some fixes. But then I guess like a late 2024, we heard about this Anthropic Cloud computer use. Cloud, if you're not familiar with, is another large language model developed by Anthropics. They are very much hyper-focused into like responsible AI. And the way the computer use is the terms that I come up with is like this. First, you give cloud or the LLM access to a tool and the prompt. The tool could be like Terraform binary. The tool could be a Python environment, bash and et cetera. And then cloud will decide what tool to use depending on the prompt. And then based on that, it will extract the input and output from the tool that's run and then evaluate whether they have to run another tool or not, et cetera. And then cloud will continue to do this, continue to calling the computer tool use until it's fully completed with the tags. Now it's kind of like very complicated. I think it's better if we just show the proof of concept of how this works. So here, I think I have a quick recording of that. Manu, you want to tell more about like how this works? Sure. So at this point, again, you can see a browser being hosted, browser being used for the interaction with cloud. So think of us trying to look at an example of a resource, providing a prompt specifically for that resource to cloud. Now we're using cloud computer use at this point, but using a browser, using VNC, trying to interact, trying to figure out what needs to be referenced. Say, for example, in this case, a data pool project. We're trying to create or ask cloud to generate an example for that resource. It works well in the sense that it's able to identify the schema, identify the inputs, create an example for it. But at the end of the day, this is a browser-based interaction. And there are pieces in there which really drew us where cloud was able to identify the right commands. If there are permissions required, it was able to repurpose, reason with it, and create those specifically for that example. Now, going back to the original thought, this is still run on a browser. It's taking up your space, it's taking up that particular amount of time. It is running on a browser, which means it's probably not that easily disabled. That is what led us to sort of looking into the next set of steps. Now, what we have in here is an example of what the demo, what the static content generates. Vali, you want to add something to it? Yeah, yeah. I think what I recognize in here, like we're seeing is like we added the initial prompt, right? And then cloud in here have access to this like a virtual environment. So it starts by opening a browser, learning about the schemas. It's opening like AWS documentations and et cetera. Of course, because this is a demo, so we kind of make it try to do a cool stuff by like opening IDE in here, right? And writing the Terraform file. But literally everything that's running on the screen is driven by the LLM itself. Like the LLM actually making a call. I see now it's running Terraform init. Like we're not running a Terraform init. It's just the LLM making a call and then using Terraform binary, right? So here I can see I'm moving my mouse cursor just to see what's the output. And I do see like it's running a Terraform validate and it find an error, for example, right? It's at LLM now start to reading this error and try and figure out how to fix this error. There's just like small syntax error. I think that was on line 22 and it fixed it. And now it's running another Terraform init or Terraform validates and it's validate that the configuration is valid. Yeah, so like when you say like it's all in a browser, it takes a couple minutes. I think this whole demo itself is about like a minute or so. But then it kind of gives us an idea, like what is the capability that computer use is bringing to us that doesn't exist previously. And of course, doing it in this kind of like a UI environment is not the most effective way. And we will talk more about how we solve that. So now let's move on. All right. So let's talk about the hypothesis we have on this scenario. So earlier we say like give LLM the schemas and the documentations. But now what we say is like also give LLM a space to start iterate and reasoning and thinking about how to write this Terraform configurations. And on top of that, we also give the LLM a tool. In this case, the Terraform validate and Terraform plan and apply so that it can continuously try to loop into and make the changes to the initial resources or example that it creates. And then only when the LLM itself or cloud feel comfortable that the whole requirements is met, then we get the final resource examples. So really the idea in here is that we provided factual fact in terms like resource schemas and et cetera, give the LLM access to tool in the secure environments, isolated environments that you can run Terraform validate and applies and et cetera, and then do a cleanup and finally provided the results. So it looks very cool in paper. We actually have some working example of this. So now, Manu, I guess now we're going to talk about the actual execution of the code, right? And before we perhaps talk more about the architectures, why don't you walk us through about this next demo? Oh, good. Yeah, before we get to the architecture, I want to show you how this works, right? So essentially, we are trying to look at a new resource called CloudFormVPCOrigin, which as you see, doesn't have an example. Look at public key, that's an annually generated example that we've worked through. Now, intention is to use LLM at this point to generate that example, being able to create in the format that we need to provide or sort of open a PR against our innervstc provider. Running an orchestration script, but what is essentially doing it is I've provided the resource that I want to create. A step function is running in the background, which essentially goes through a certain set of steps, which we'll discuss in the architecture diagram. It goes through a space of creating it, validating it, reasoning with it. And once it succeeds, creates the necessary artifacts that I need from my side for opening a PR. And in this case, I've gone in with the steps of introducing or showcasing where it downloads from, what the output looks like and things like that. Let's say what we need is an example of the resource and the template that we have. In my local machine, if you see the Terraform resource that is generated, it is not just generated the resource that we asked for, it generated all the subsequent or related supplemental resources that makes it a perfect, complete example for the user to use. Now, along with that, if you see the VPCs, the Elastic Load Balancer, anything that is required for that VPC origin to work. And we had a template that was already earmarked for it, and it actually creates it based on the prompts that we have. And it creates the necessary template that is required for the website to render. Let's take a look at step function in the console. If you refresh it, pick up the latest run, it see that succeeded, you have one iteration of that run. Again, we'll get to these steps in a few seconds, but again, I just want to show that the step function is going through a variety of stages or steps while it reviews the state and goes to create those artifacts for us to be able to publish this PR. If we get a state view, the final artifact is essentially the target resource that we asked you to create. In this case, AWS CloudForm VPC origin. Go back to the S3 bucket, which is using as an artifact, refresh it. We get something which is by the ID, the hash sort of matches the execution ID. And we have the resource creator, which is matching the resource that we created or requested from Cloud at this point. Now, this is the examples. The output in this case is sort of a way that we have, we expect our provided examples look like. So this is a PR, active PR that we have on our AWS VPC product example on our repository with the list of additional resources that we are able to create. And with that, that's the demo. But let's take a look at the actual architecture diagram. Let's look at sort of view some of the key components there. Yep. So Manu, before we jump to that, I think one thing that's really cool is that what you just shown is like, we take the initial demo where you see the browsers and the chat systems that kind of orchestrate. And you break it down into like a Lambda function and also like a step functions in here, right? And then you have an orchestration script that allows you to execute that. So I'll let you dive a little bit deeper into this architecture. Sure. So what you saw in there is essentially, the script that we did was essentially calling the inference workflow. Now, at the heart of it, there is a step function which orchestrates some set of steps, which allows it to interact with Amazon Cloud through Amazon Bedrock through specific instructions. And Vali will talk through the system prompts and the specifications there. Now, what we have in here is an iteration of certain set of Lambdas, which is already been, these are containerized Lambdas. So what we have in is created a custom image with all the necessary tools that are required for card to function. But we have a few steps in there. Creation, as in the create Lambda essentially does the initiation of it and creation of the resources. And it also deletes it. But the delete step is in case the previous step timed out. Again, we have a 15 minute timeout from the Lambda standpoint. There could be resources which requires more time to create and delete. So you want to keep those in mind. We have those iterations of Lambdas built out in there. Create, delete, review, clean up, summary and clean up. So one of the other one, for example, there are steps or stages built in to make sure that we're cleaning up the resources once it's created. And we effectively look at the summarization of what we created. Let's say, I created a CloudFront VPC origin. I am asking Cloud at that point to say, hey, can you generate a two sentence summary of what we created here? Along with the example that we have. And if you see that iteration, finally, if the entire thing is created, depending on the file markers, for every step, every Lambda at this point creates what we call as a file marker for creation, deletion, review, clean up, summarization, all those. Once it's there, we can take the artifacts that are generated, saved off a simple S3 bucket and use that to generate the examples that we need or copy the examples of the artifacts that we can use them. So essentially, what we have in here is a step function workflow with an iterator. Currently, I believe the iteration is set to a default of one. You can always change it. We are using a custom image. The Docker file is also available for us. From the, and essentially working with Cloud, applaud using Amazon Bedrock and saving those results that we generate from these in the format that we need into an artifacts bucket. Very cool. Well, why don't you walk us through the prompt that we set to get us here? Yeah, yeah. So very cool. I guess I want to also highlight a couple of things in here. Like you say, Manu, like the Lambda function, which is kind of like a container to run your serverless, also includes the Terraform itself, right? But the LLM that's been executed is actually running on Amazon Bedrock, which is kind of like a managed service, right? So you don't host your own LLM. In this case, you're actually interacting with the Anthropic Cloud that is managed by the Amazon Bedrock. So you're only kind of like using the LLM when you need it. But in some of these phases in here, I think you mentioned like the cleanup breaches, it's not actually have to use the LLM itself. Very cool. So with this understanding, I think let's jump a little bit deeper into like the prompt itself. At a high level, I think we talk about like the architecture that Manu explained earlier, and we have set up some sort of a system prompt, which is like every time any of the LLM, whether that's to create the resources or to delete the resources, will have this part of the system prompt. And this system prompt basically adopted from the initial quick start of the Anthropic computer use. So we take the inspiration from their code and then kind of codify additional rules in here. So basically what you do in here is you tell the LLM, you are currently running instead of like, say, Mac OS X. I realize you haven't changed that. This is actually now become like Linux if you're running it in like a Lambda, right? And then we also kind of explain like, hey, do not install any new applications, like use what you already have, such as the tool commands, such as Terraforms and et cetera. We also kind of be a little bit more peculiar, say like the current date format is in this format, for example. And then we tell the LLM, again, this is still a system prompt, if you need to find AWS-CC resource schema, here's the link to, for example, the registry. And then if you want to look for any equivalent of that, maybe in CloudFormations or something else, go to AWS documentations to find that. And then we also set certain rules, because AWS-CC has some nuance, for example, the tags. As you see in the bottom part, there's a specific format and how you specify the tags. But also we kind of give them a guidance, like be mindful about security, make sure you set up like a least-privileged permissions when you set up the permissions, and also be mindful about certain resources. Certain resources will take longer to create like EKS cluster. And in that case, maybe we want to just offer input variables. And we kind of let the LLM use this governance rule. And then on each of those, like a unit that you saw earlier on Manu's screens, when he presented the diagrams, here we have the user prompt when we are creating the resources. Kind of give like high-level objective of what is the resource that the LLM needs to create. Couple rules to follow, right? Again, this is like specific rule on the like during creation of the resources. And then essentially the sequential order of how we want an LLM to operate, like navigate the working directory, download the schemas if necessary, running some like a CAD to just kind of read the schemas, running init, Terraform init and validates. Finally, like a running apply and also destroys. And we also tell the LLM to set up some marker. And this marker is important as you want the LLM to navigate from one state to the other, such as if created marker is completed, then go ahead and maybe proceed with the other process like you saw earlier on Manu's screens on the diagrams. So we're going to open source a lot of this sample in here. So just kind of give you high level as we go to inner working of how the sample prompt works. So now with that, let's talk about the results. And I'm super happy with this. Like we did this run for like three days and takes three days because it's a holiday. We run the script, we go and grab coffee, meet with family and come back during that whole Christmas holiday and keep kind of like running it. And now for that three days, we are able to generate about 450 resources. And it take us about like 400 bucks in terms like the bedrock inference cost. I don't want to belittle like a lot of the human effort that has been made since 2023, but just going to give some contrast in here. Myself, Manu and couple other contributors working in early 2023 until end of the 2024. We have about 250 resources that we created and takes about two years. Of course, we're not doing it nonstop, but between our side job, our second job, our first job in AWS, those are like at a time where we're able to allocate to do the contributions. So it's like a very night and day in terms like how much we can accelerate this whole process. The other thing is like we are able to reduce the hallucinations in here by providing the LLM with tool access and then also allows it to kind of really validate its own works. And as I mentioned, human is always in the loop. We don't let the LLM to make a pull request. We still want to be the one for control. We supervise, look at the results. When we're happy, we make a pull request and we have human to human interactions in that pull request. There are other example of use cases in here. Manu, you want to maybe expand a bit about this potential use cases? Sure. I mean, as a new resource, as a new resource is added or a new schema gets published in the CloudFormation registry, we will be able to use something similar to validate the resource prior to release. So you think of creating a resource, an example, even before the schema is added to the registry made available to the provider. Even looking at existing example, as we iterate on the schema, as new features are added, we are able to validate that existing examples are done without really having to have a human spend a lot of time understanding those schema changes and verifying all those things. And why not extend it to create tests on any small, let's say, even go ahead and open pull requests on our behalf, if that's the case. Awesome. Yeah. And also we can explore like, in fact, as part of our whole process, we find some resources that we are not able to fully generate. And we actually have LLM to do additional troubleshooting for us to kind of give some reasonings. We're going to report more on that in the future. But for now, if you want to try this, we have a link to the AWS samples repository where you can find the sample codes. And hopefully that also will inspire you to do similar experimentations, maybe with the other provider as well. And with that, we want to say, thank you so much for joining us with this HashiTalk. You can find us on our LinkedIn as well. And also maybe look forward to see you all in the different, maybe future opportunity. Manuel, last word from you before we stop. This is, again, this is just an example of what we can do with an LLM in the infrastructure space. I mean, the ceiling is too high. So take a look at it, open pull request against it. If there are changes that you want us to incorporate, if there are ideas that you have, please let us know. Yeah. And with that, thank you so much. Thank you for joining us and enjoy the rest of the HashiTalk.
