Transcript
or a couple of days before, and I can't. So, before we dive in, let me ask you a question. If you can, by a show of your hands, how many of you feel that AI hallucinates so much that you can't just trust it for your work? Oh, that's quick. Okay. I'm Manoj Chandrasekhar. I'm a senior delivery consultant with AWS, and I have... I am Dirk Avery. I am an engineer on the Terraform AWS provider team, and glad today to be here with you for this presentation. I want to make sure you're in the right spot. This is paying homage to our AI overlords. Wait, wait. We changed the topic a couple of weeks back. We changed it? We good? Oh my gosh. We good? Well, I didn't prepare for this, but... Well, back to the topic. Well, can we all... We're trying to talk about whether, if and how we can get AI to work for you. The short answer is, yes, we can, but to reliably. All you need to do is invest in some techniques, methods, and maybe tools or mindset. Again, before we get into those things, let me take you a few months back. I was sitting in my study, I was working on a module called Terraform AWS Glue. Anyone not familiar with AWS Glue? Glue is an AWS service which lets you write ETL pipelines. ETL pipelines. Now, I had to update it to support Glue version 5.0, and I broke things. I wanted to fix it, and I realized I didn't have enough tests. I didn't have any tests. How many of you like to write tests? I do not like writing tests. Oh, that's not what I expected. Okay. So, I was looking at some way to make sure it's easier for me to write tests for my Terraform modules. I do maintain a few more. I looked at generative AI for many other things, so why not look at this and try to create an approach where I can follow and scale it across my modules. Let's see how I did. So, what you're seeing here is, well, it looks like a domino. It's actually in the phase that you get after you invoke Amazon QDeveloper. We'll get into those a little later. Bear with me on it. All I'm asking is, can you create some Terraform tests for the module in this directory? And that directory is my AWS Glue module. For me, if I'm working with a module or any set of resources, I want to know what is in scope, what else is in scope with respect to the resource, because they don't usually stand alone. They love an IAM role for sure. Some S3 bucket somewhere, VPC connections, if it's all that. So, the first thing the agent in this case is doing is to find or get the resource inventory, because that's critical. The next thing in this bit is to identify the Terraform version you're using. If you're using something before 1.5, you don't have test. This framework doesn't support test. And I'm on 1.13, I believe. That means I have parallel execution. Why is it important? Well, I have tests which might be independent, which could use that time to get the test running quickly. So, that's good. I've got those two ironed out. What do I do next? I wanted to create a folder structure, like a test folder, so I can segregate those tests in that. Now, as a human, I would have run like one test, verified and done that. But my agent has actually gone ahead and said, I'm going to create 17 tests. Why not? Now, the best bit here is I'm going to use Terraform tests. What do you do when you write a test? You want to validate it works. The easiest command to do is Terraform test. That tool use, we'll focus on again, is important because it helps you to validate the inputs that you've given. In this point, it's just the run blocks that you've created as part of your test. Well, do you expect everything to work? Maybe, maybe not. It did not work the first time around. It failed, for sure. Now, that tool use is essential and important here because it gives out feedback back to the agent I'm working with. So, it's looking at the test and say, oh, this test is failing. Out of the 17 I wanted to create, four passed, one failed and 12 skipped because of it. Now, it's trying to go through the motion of doing that again and again. It's just that loop. You as a human or as an agent is trying to figure out what can be done to get to the final goal, which is you want successful test. You have 17 tests out there which you want to get to success state. Let's see how it goes. Well, it is done. That was about five to six minutes of me speeding it up. Now, along with that, there are some instructions in there for me to the agent that I've given to give me some documentation around it. Now, it talks about what resources there are and some structural constructs that it created. Maybe, me, I want to run the test again just to validate. Well, it works. Let me take a look at the actual documentation and the test it created. The README that you'll see, it has a list of what we talked about, the list of the resource inventory that it had from the module that I created. It does have the other constraints that it had, the Terraform version, anything else that it used to create those tests. One thing which I didn't add initially, but it came up again is, oh, there was a test that I tried, did not work. Maybe go back and verify why it did not. It's more a pointer to me to verify why my module is not testable for that regular one. It's a module, so there's conditionals in place, so many things. Going back to the test itself, you see the parallel test block, which is either global in this case. You could do it per run block as well. I have set it to just plan. You could always extend it to applies. On the basis of that, I could just create tests with some reference context and the initial prompt saying, can you go create a test for me? Manu, from a very simple prompt, you got 17 tests. We're getting ahead of it. Let me get to it. That was my prompt, right? I'll get to the context of what I gave in the background to get to this right. What we learned today, we will look at Amazon QDeveloper. We'll dive deep into what capabilities exist and techniques exist that you can use to be more productive with AI. What we saw was a practitioner problem, writing Terraform tests or any sort of that point from a module standpoint. I'm going to hand it over to Dirk later to talk about the bigger problem. There are no resources until it's in the provider. You don't have any deployments. You don't have any workloads. You don't have any business value until then. Let's see what Amazon QDeveloper is. Even before getting there, AWS Generative AI stack. I have to put it up there. For any type of workload, depending on what your generative AI workload should be, there are different services that cater to different things. Whether it's to build models, train models, or to generate insights, we have specific services for it. We are focusing on Amazon QDeveloper here, which is purely on the software development lifecycle phase. Now, where can you find it? Wherever you work. It's in the console. It's in your favorite IDE as an extension, or as we saw earlier in the CLI, like if you call CLI your home. Now, you talked about something about, was it just the prompt that I had there, right? Now, one of the challenges that I feel with any LLMs, any models that you work with, is it's trained till a certain point in time. It doesn't know my style. It doesn't know my organization style. It doesn't know how I wanted to create. If I want to collaborate with, say, another colleague of mine, we follow a certain language. How do you do that? Enhance context. You have multiple ways to do this. The simplest way is to use the slash context option within QCLI once you invoke it. Now, how it helps is it allows you to create either global rules or global markdown files, which can be instructions to do certain things in a specific way, or it can be project-level rules, which means I have this in a specific folder in the project I'm working with. Why? Because it helps you to get that additional information into the model that you're working with, which may not be aware of till that point. Now, let's see what context I was working with in this case. Single prompt, huh? Not really. So, I didn't, in fact, call an agent. I was actually calling an agent by default in there. There are a few sets of documents. I was using a trick there to use a folder called rules inside my current directory where I was working to push in some additional metadata, some documentation. Seven markdown files? Well, start off with a testing strategy or testing master reference saying, hey, this is how you structure your test. These are the things you need to look for when you're creating patterns. When you work with mock providers, if your type of emotion is above one, there is parallel execution possibility. So, why not look at that piece? Well, then, what do you call, how do you finish the whole thing? How do you call it successful? There should be some criteria for it. And the one that I go back to all the time, troubleshooting. That document I've probably kept in every context documentation available because I keep going back to it, whether it's Terraform test, whether it's Terraform runs, validation, anything of that nature. So, I know it's a lot of work to write all of these context files. Did you write all these? I started with a single prompt saying, can you go create a test for me? And I ask you to reflect on the past conversations. I'll say that again, reflect on the past conversations that you had and enhance the prompt. So, you don't have to create all these things yourself. You can start with a simple prompt which works. Give it enough information based on your style, which you might not know. Then you interact with it. And in the conversations, it can refine that prompt to the way it wants. I split it again to seven different documents because I felt there are certain things which make sense for other use cases too. You may not need to, but that context and the prompt that is required, you can build it on the basis of the reflection. Now, why CLI again? On top, I prefer CLI more towards, it has some additional tools available. It has access to the file system. I have access to all the terminal tools, Terraform binary, Sentinel binary, anything that you can have access to from your terminal itself, which means you have the ability to run that tool as a validator, as a feedback provider to your agent. So, why not use it that way? I agree with you totally. I'd much rather use the CLI. I've tried it in the IDE, but it just works better for me in the... That's why we're on the stage together. Now, it'll be a mess if we do not talk about MCP in any conference these days. I don't need to explain what MCP is. Again, it's that extensibility that your agent demands or needs to get the work done. It's another layer of another way to provide context, whether it's your reference, whether your internal registries, Terraform registry, AWS documentation, anything that you want to provide that additional context to your agent for that particular thing. Two things to keep in mind. There's so many MCP servers. You don't need all of them. You just need some for those specific tasks. Pick and choose what you need for your task and don't overload it. Some do require credentials. So, try to find ways to securely provide them. Environment variables, if you pass in the MCP settings or JSON to others. Some way to not show it's in that plain text in JSON file. Cool. Now, let's wrap this thing up. Agents. For the keen eye, you would have seen when I showed the context earlier, it was showing the default agent. Q already has a default agent, which is using the rules from specific locations and even the readme that you have in the document in that particular project. So, agents are a way to create customizable agents for your use case, whether it's Terraform test, whether it's to scan something, anything of that nature. Why? Because it allows you to show and hide the usage of MCP servers. Any tools, trust any tools, which I forgot to mention earlier, you can specify or trust tool A, but not trust tool B, because some of them are read-only, some of them are non-destructive, which you're okay with all the time. So, agents are another way to reuse that context. Again, context is the king here. Quick examples. Two tools, two sets of agents that I have in there, Terraform test and scanner. You can see the different sets of tools that I have in there. Different set of MCP servers have access in there. They don't need all of them. Similar context. I mean, they look similar, it's just that the underlying documentations are different. They are referring to three different ones because test doesn't need everything that scanner has. That's great. You've got your agent, you've got your context, everything's working fine, but there's still something missing. You. You, me, everyone. Well, I still believe we are the context from our domain experience who can, well, it's your creativity and it's your domain experience which drives the solution that you're building. Your style guard, your organization's needs. That is being documented in a certain way as context for the agents to work with. So, if you can do something, it is to understand how context works, master that piece of it, use rules which sort of reduce the cognitive overload. Say someone new is coming to your team, you can reduce that overload and someone new coming in with these rules that you can use QCLI to help create those things quicker. Use agents for those specialized tasks that you need. And I can't stress enough, reflect, talk to an agent, you know, might seem silly, talk to your agent about that last conversation that he had. What worked? What did not work? How can we elaborate and refine our context on the basis of that? And then refine it again. It never stops. I think that the documentation that you create or the context and the rules are actually great for humans as well. I mean, it's not just the AI. Yes, it is. Yeah, that's a side effect of it, right? You're actually creating documentation for someone to follow too. Now, I would have said create the implementation plan, but right now QCLI does it by default. I mean, to an extent. If you are on the latest version of QCLI, it sort of breaks down the overall task into smaller chunks so that you are able to run with those and, you know, create them to resume or create that sense. Cool. So, what can you do? We talked about some practitioner stuff. There's a lot more on the provider side. I want to hand it over to Dirk to talk about the bigger challenges. All right. So, as we said at the beginning, Dirk, I work on engineering in the Terraform providers and at HashiCorp, and I am super excited to announce while we're here that the AWS Terraform provider has reached 5 billion downloads today. Nice. And it would not be okay to announce that without giving a huge shout out to the contributors that have made that possible. So, thank you very much, contributors. Please give yourselves a round of applause. We really appreciate it. I also wanted to give a shout out to my family. Hey, fam. No child left behind. They'll know what that means. All right. So, let's move on and make some of this work. So, Dirk, we're going to create something today, and I'm excited to be with you because we're going to do something a little crazy. So, when you're an engineer, creating something useful that people actually use is very fulfilling. I love that. And so, we're going to create something new, a Terraform action. But here's the catch. What in the heck is an action? Terraform is declarative, right? You describe what you want, it goes and makes it. An action is doing something. So, let's make one. A way to publish a SNS or a message to AWS SNS from Terraform. But the problem is, there's no action documentation. There's no um, the model wasn't trained on it. This is brand new stuff. It was just announced at this conference. So, it might sound foolish, maybe even a little bit dangerous. But what do you think? Should we try to do something crazy? Team up with Q and try it? Why not? Nice. Let's try it. So, I'll take your silence as indication you want to go ahead. Well, here we have a blank canvas. And we're going to start and ask Q for a plan. Because this is kind of a complex topic. And so, I give it a plan or ask it to create a plan. Tell it what I want to create. And then I give it, I point it to an example. Another action that I wrote. The lambda invoke. So, I also give it a small context file so that it can keep track of, you know, just some of the things I've thought about. And off it goes. It's running. I'm not quite as trusting as Manu. I'd like to do a lot of yes, yes, yes to let it go ahead. But here we have a five-step plan. The actual implementation, registration. Got documentation and tests and a generation step. So, we're going to head off with step one. Create the actual action itself. So, it takes a couple minutes. And boom. We have got code. About six minutes into the process, we have got an action. Which I think is pretty cool. So, I scroll back because I've got to remember what the steps are. Having a plan keeps me and the AI both on track. And it finishes step two, the registration. Now, we're going to move to step three. And create the documentation. I don't know if you love writing documentation, but I don't. So, here we're done in eight and a half minutes. We've got the code, registration, and the documentation done. So, now we're going to move to step four, which is critical because it's the tests. And, you know, if you're a little bit uncertain about AI doing the right thing, tests are a way of verifying that it's done the right thing. So, I ask it to create an SNS topic, an SQS queue, and then a subscription and a poll to make sure that I'm getting the right, you know, getting that message off the queue. So, remember this for later, but it does what you ask, not necessarily always what you mean. But it goes out, it creates the tests, and check the plan. And in 13 minutes, we've got steps one through five done. So, this probably means it's all good to go, right? But we're going to try compiling it just to make sure. And it's got some issues. So, we're going to enter this iterative loop of run, fail, fix, run, fail, fix. So, it's trying different compilations and fixing each issue as they come up, which I love because Q is never tired and always happy to continue on with each step. Definitely not my next son when I ask him to put the trash out. Yes, and sometimes co-workers can get cranky if you keep correcting their work, but Q never does. All right, so, it now compiles successfully, which is super awesome. But now it is time to make the sausage. While he does that, he secretly didn't tell you that he's actually running an agent out there, if you noticed. The framing is slightly different. There's an action running in the background, agent running in the background, while he initiated a queue. Yeah, yep. So, we tried running the end-to-end test. I didn't have credentials in there. I fixed that. Now it's having trouble with types. And so, we're going to, it said at the first stop, we're going to, it said at the first option was, hey, just remove that argument. I'm like, no, we can't do that. So, let's try to make it work. And then I point it to some examples of other types of places where we do that sort of thing and show it that it's possible. And it takes that into account and goes again. So, we got run, fail, fix here, which is a fun pattern. It actually works really well. So, it then tries to do manual data conversion, which is fine, because it doesn't know we have a library for that called autoflex. So, now we've fixed two issues. We're moving forward. It's using our library, and we're going to run the test again. And five minutes later, nothing. I did say challenge. Yeah, it just hung. So, I aborted the test, and we're going to try and see what is going on. I went out to the AWS console. Let's look out there and see what's going on. All the components are there of the test. So, I don't see an issue. But then I look through the test file, and I see that there was never a trigger. So, it was never going to run. So, Q happily adds the trigger. And now, we're going to run again with the trigger. But five minutes later, still nothing. So, this is 51 minutes of real time, and it's not working. But after this run, after I have to abort the test again, we figure out that there's a new culprit. But IAM policy didn't have a version. So, I kept thinking the policy was not updating. So, it wasn't even hanging on the action. It was the IAM policy. But anyway, we got it working, and we send it out. And now, we've got an unsupported argument. So, it's going to go back and forth and do some run, fail, fix. We get to a new error, unsupported block type. And it's going to try it again. And I'm just hoping, come on, please work. And it fails. But at least now, we are getting out to AWS. So, we're making progress. We're out at AWS. We're at about an hour in. It says that we've got malformed input. But we're getting there. So, I'm going to run again and see if Q can fix this issue. So, I give it a little redirect on how it's using the data conversion library. The data types are just not working between the schema and the model and the test, the way it's trying to use them. We've fixed that. We're going to say, hey, try it again. Will they work? Come on. Come on, test. I know, right? But failure again. So, we're an hour and six minutes in. It suggests, why don't we not use the library? I'll do manual data conversion. I say, I know that our library can handle this. Let's try it. And Q never cares. Like, if I do a control C and interrupt it and say, hey, let's go this different direction, it's like, okay, let's do that. So, I redirect it, give it more examples, and hoping once again that it will work. And it fails again. So, we're an hour and ten minutes in, and we are fixing issues. Sometimes it feels like a greased pole because we climb up a step and then slide back a little bit. And here, I steer it wrong. I actually give it bad information. I forgot that when you use map block key, which is a way of doing maps as lists, that you've got to have an element in the schema. So, I told it, you don't need to. And then I figured out, finally, oh, yeah, you do. And told it, you know, that I was wrong, gave it good information and some examples. And now, it's going to try again with good information. It's very hard to make things happen if you're given bad information. So, we're an hour and a half in, and the struggle is real. So, do we have a resolution? So, it asks if I want to run all the tests again. And I sit there for a minute, not sure. I can take the stress. We run the tests. Come on, tests. Will you work? I'm, like, staring at the screen. Seconds start to feel like hours. Like, please work. So, this is the moment of truth. I've got my hopes. I've got my dreams. Just pleading for it to work. Sometimes, I feel like I'm going to stare a little hole in the screen, waiting for it. You feel the nerves. I'm, like, come on, work. Any minute. Give it a second. Give it a second. Oh, my gosh. Let's go. Get in there. We see the word pass. The most beautiful word ever invented. So, sweet success. After an hour and 33 minutes in real time, success. So, I am very happy with that because in an hour and a half, I have got the code. I've got tests. I've got documentation and verification that it works. So, we look at now we're seeing code that didn't exist just before this. Looking at the documentation, it's gorgeous. And we've got a pull request, and it's ready to go. And so, here's the truth. AI, this is the process. This is the whole thing sped past us. So, you saw that there was a lot. We were back and forth. And I was staying engaged. But AI is not the hero. It's the Chewbacca. And it's the Samwise. And if you're SpongeBob, it's the Patrick. So, you've interacted with Q and AI agents in general. How do you think it has changed you when you work with it? It's changed me because I've become a better communicator. I have to be clear and concise. I have to figure out the right amount of context, not overwhelm it and not underwhelm it. You are the context. Yeah, exactly. So, we also break up things. Instead of just saying, hey, do this huge complex thing, I've also learned to ask for feedback. Instead of just saying, go do this, say, what do you think of this? So, in the future, people worry that AI will replace engineers. But I don't think it will. But engineers that use AI might replace engineers that don't. So, you are the hero. And you're going to be able to do more than we ever imagined with. Good segue. So, the only constraint that you have in here is your imagination, right? I saw an example where I had examples. I saw an example of a demo where you had examples. This was completely new. The whole construct is new. You can provide tools like Golang to use its own native validation to give you feedback to get to the next stage. If you feel there's gaps, bridge it with additional context. And so, what do you build? I urge you to download QCLI and give it a try. Start small and experiment with something you want to automate from your side. Build your own prompt library. Your organization may already have it. Collaborate it. Contribute to it. And if you're like me and was interested in MCP, stay right in. Because you have a session right behind you, right behind us. Gautam was talking about automating secure DevOps with hashing of MCP servers. Again, the context and the prompt that we use are right up there in the QR code. Thank you for being a wonderful audience. If you have any questions, find us outside. We'll be right out here. Thank you.