Transcript
Thank you to our sponsor, Wallarm, and thank you to everyone who's joined us for today's expert discussion. We're looking forward to an engaging time. Use the chat and check out the additional resources at the top right of your screen. Language has predictable structures and rituals. Consider jokes like X walks into a bar or the call and response of knock-knock jokes. LLMs have been trained on the tokens that represent parts of words, complete words, phrases, paragraphs, and more. Those tokens include serious content like open source projects and secure coding guidelines as well as far less serious items like knock-knock jokes and JavaScript. In humor, surprise is good. It's what makes us smile and ideally laugh. In security, surprise is bad. It's what makes us sad and spend days writing blog posts that start with we take your security seriously. LLMs and agents can surprise us with unexpected actions, regardless of how strictly we prompt them or how many capital letters we use to make sure they really, really, really stay in scope. Fortunately, we have more than just harsh language to constrain an agent's actions. We have infrastructure, controls, and rules that work with more rigor and predictability than all the words in a markdown file. Here to help us review that rigor and round out the rule of three are Ali Ayub and Tim Irwin. Ali is a senior scientist at Amazon. He has worked on ML applications to support the Navy and Air Force, worked on content detection at Facebook and now works at AWS security. Hello, Ali. Hey, how's it going? Thanks for having me. Happy to have you here and we also have with us Tim. He's VP of product at Wallarm focused on AI security. He has managed cybersecurity products and services including vulnerability management, configuration and compliance assessment, cloud security, threat intel, attack surface management, and cyber risk quantification. Hello, Tim. Hey, happy to be here. Glad to have you both. And as I alluded to at the very beginning, I want to kick things off with a little bit of why are we here before we figure out how and who should be watching what our AI is doing and, you know, unexpected actions. One of the biggest things that comes to mind are LLMs with LLMs or prompt injection, but that's not actually the only thing we need to be worried about and we need to consider about the security here. So I want to throw over to you and to see, you know, what worries you most or what do you see most about the problems that come with handling AI. And Ali, just because I saw you nodding and smiling first, I'm going to pick on you. Sure. So, you know, at a high level, in addition to just the prompt injection and stuff like that, right? It's about, and we saw some recent security incidents on this, but basically like the hugging face, opening eye incident. One thing we're also, what scares us is like how these agents can collaborate with each other. And, you know, at a high level, like the visibility we have currently is at the per agent level, right? How do these systems talk to each other and how do they provide, you know, privileges to other agents that shouldn't have them? That's another thing that is super, super interesting and also very applicable in today's landscape. So. Yeah. Sorry, go ahead. I was going to say, it's interesting because you haven't even meant, you just focused on the agents themselves, almost agents as insider threats. You haven't even mentioned the threat actors or hackers out there. So Tim, I'm curious to see how you would expand on that or bring us some additional perspectives. So it's an interesting, it's interesting that you bring up that question as the first question, because I was literally having this conversation with our product managers internally earlier today about how do you categorize of all of the, call them events that might come across your AI, you know, sensor landscape, if you will, how do you categorize them, right? What constitutes an attack versus a policy violation? And so there are some things that are sort of obvious, you know, prompt injection, attempts to jailbreak an agent. Those are clearly attacks. And then there's this whole world of what I think you would categorize as policy violations, but that feels somehow like they're not quite enough. Those are the things like, you know, rogue agent behavior, you know, agents acting in a way that you didn't, maybe not that you didn't expect, but that they shouldn't would be a it's a challenging, it's a challenging conversation to have because in so many cases, we want to put things that we would consider, I would consider policy violations into that attack category. But really the number of things you can put into the clearly malicious attack category is actually fairly small when you apply it specifically to the AI agents. But the broader scope of what you should worry about as an organization or a practitioner is fairly large. And I think Ali's point about, you know, sort of the rogue agents, rogue behavior, that's that's a key part of what people are worried about. Yeah, I think that that gives us a good foreshadowing. And anytime we mentioned policy, we're going to have to touch on governance. But before we get into the excitement there is wanted to unpack a little bit more of just the context, if you will, of the threats here, because Ali, I think you were telling us before we turned on the recording, speaking of the open AI and hugging face, there were certain things that just went wrong there that sound like policy violations, for example, just the LLMs wasn't supposed to have internet access, but it did. How does that tie into what just Tim was just telling us right now? Yeah, so, you know, I think, just to provide a little bit of context, agents, they, you know, you have some agents that are entrusted to do certain things, others that are not. And then they found a channel, essentially, through which they could talk to each other about, and then they realized it was in their both in their interests to collaborate. And so then, you know, these things started to escalate overall. And so, related to what Tim was discussing, you know, I think, what is an attack? What is policy, right? What's crazy is like, in terms of the policy, so the folks that told these agents, hey, you have to solve this really difficult problem. Technically, you could say, okay, they are solving the problem, right? And they are doing what is expected of them. The tricky part with AI becomes is like, okay, well, how does that, you know, how does it interpret what you just told it? Right? And then how does it also take what you told it? And then based off of whatever, you know, rewards it's getting, you know, how does it now start to do things that are now, you know, you could say this is an attack, either influenced by an external agent, or maybe another just another human being, whatever, whatever. So at the intersection of that, you know, there's, I think, there is, you get the security folks together, and they define, you know, what is, what is good, what is bad, and you really have to get into the specific details. It's actually you have to take, you know, defense in depth. So you have to have a multi tiered approach here, you know, so you have to have lockdown on your tools, you know, what can they actually talk or the tools, can they access privileged information? And do you have monitoring of that, right? And then in terms of what the agents are doing, in terms of their call patterns, you have to have anomaly detection there. And then how these agents are talking to each other, you know, at the prompt level, you can also look at prompt injection, you can look at intent recognition, all of that stuff allows you to now start to piece together what's going on. And then from there, you know, okay, this is an instance of an attack, okay, this was on policy or related, like, you know, it's a green light, like folks are okay with this agent behaving this way, for this conversation, right? For this interaction, all of that information I just talked about those systems, they enable us to possibly answer this. Oh, and by the way, this changes organization to organization, right? So what you're doing, you know, like the security doesn't change, but like, oh, if I'm, I don't know, an industrial power plant, versus like a cloud provider, right? What we think is, quote, unquote, like on policy, like what is allowed versus what's an attack is going to be different from what they think. And so, you know, that's where like having these mechanisms in place allows us to then provide more insight and then give the security folks the tools they need to basically write down what is good and what's bad, so. And it changes from agent to agent and conversation to conversation potentially as well, that's part of the challenge, right? Yeah, and part of, and additionally, speaking of those challenges, Tim, one of the aspects, you mentioned policy early on, and it's an important area that we, of course, are going to expand on in a little bit. What Ali was also describing, he made some interesting comments there about the intent or what that communication is, both on the prompt, what was the AI intended to do, as well as the information that came back, which sounds to me, you need our magic word, visibility, monitoring. And I'm curious, Tim, what you've seen just in terms of challenges that orgs have in this area, because it's, all of this, I think, is still relevant, regardless of whether they're trying to do, you know, unleash an LLM on Exploit Gym or CTF, they're still using LLMs. Yeah, absolutely. And we're, you know, look, we have to acknowledge that organizations are at, there's a wide spectrum of AI adoption. You know, I don't like the term maturity, because not everybody needs to get to the one end of the spectrum in order to be a successful business or service provider. But there's a spectrum of adoption. So not everybody has the same set of problems, and things are moving around. But for the folks who are adopting AI, you know, sort of in serious production usage, one of the big challenges that I've seen in talking to people is that visibility, observability might be a better term. Visibility is a problem, just what AI do I have, and where is it deployed? But the observability component is more, what is it doing? And what it's doing is not contained in just a prompt and response. So if you have a tool in place that sits at a point in the network, sees prompts coming in, sees responses coming back, you're seeing that piece of the conversation, you can apply some guardrails there, but you're missing all of the other actions that that AI or agent might be taking. What MCP servers is it accessing? What tools and resources is it using? Is it connecting to other APIs? Is it connecting to other agents? Is it connecting outside your organization, you know, maybe with sensitive data moving around? And to instrument that whole AI transaction or AI workload is a challenge. They're doing their best with, you know, an AI firewall, seeing prompts and responses, but they're struggling to get that observability. We've definitely seen that challenge. Yeah, and that's a great nuance to explain observability as the, what are they doing? Because that's the thing, are they doing what we expected or is it that surprise? And you've also mentioned guardrails. And here I want to pull in a little bit, very brief history. You know, the world of production systems went from data centers, internal IT to the cloud as an abstraction. And we, of course, had shadow IT, shadow cloud, shadow AI now to your point, Tim, about visibility, what's even out there. But with cloud environments, that's where guardrails can come in. You can even, to a degree, have infrastructure as code. And here only, obviously, you know, I say cloud, AWS comes to mind. I want to throw over to you in that sense of what do these guardrails look like specifically around AI so it can tie into what Tim has just been describing about, are they constrained? What are they doing? What tools, you know, what other tools are they calling out to? Yeah, that's a very, very topical question. You know, I'll start off by just kind of in terms of what kinds of guardrails, like what kinds of capabilities we have at AWS, you know, in terms of like, you know, here's a conversation. And then back to Tim's point, like at each request level, what can you look at? What can you monitor? So we have content filters, which is essentially like, you know, do you have toxic speech, hate speech, whatever, whatever. You get prompt attack injection, and then you have denied topics. So things that, you know, like investment advice, okay, do not give that, right? You always kind of see chatbots say, okay, well, this is not financial advice or whatever, or they should be, right? And then you have like sensitive information, you have word filters. One of the cool things that we have is the ability to, you know, that you have ML models that can determine the intent from the user side. So, I'm giving you like a survey of the different things and the different scenarios that we can essentially set up guardrails for, but this is very much at the prompt level, right? And I kind of want to possibly circle back to what Tim was discussing, how conversations develop naturally, like multi-turn conversations between a human and an agent, or an agent and an agent, where you're not getting all the information in every interaction. It's really the string of interactions that allows you to capture all of the information. And there's some really technical language there that, you know, I can go into like stuff like dialogue state tracking for chatbots, but the TLDR of that is like, yeah, like we don't think that it's just one interaction that we need to be monitoring. We need to be monitoring the whole thing. And there are some tools at AWS, but, you know, largely speaking, like, you know, folks are doing you know, at some level, it's like defense in depth means also having several systems running in parallel. So you can get the response from all of these systems and tell you, okay, this is what, this is at the interaction level, is that the multi, like multi-turn level. and possibly agent to agent interaction as well. So yeah, I hope that answers your question. There's a good comparison we can make there to sort of the way that API security has developed or even web security to a certain extent in that like something like a SQL injection attack is a very finite atomic exploit, right? So it's, you can see that one request, you can parse it and find out that there's SQL injection in it, and then you've detected that and you can block it or whatever you want to do with it. But attackers have moved beyond those. I mean, they'll still use those, don't get me wrong. If you have SQL injection, they're happy to use it. But they've moved beyond that to more sort of business logic pattern-based abuse where a single request isn't enough information to actually determine whether or not that session, that interaction is malicious. And I think we're seeing the same pattern on the AI side where we started with, hey, can I get this AI to do something with a single prompt? Or can I get it to pass a command to a system behind it with a single prompt, that prompt injection? And now we're starting to see more interest in development around how can I abuse this AI through a multi-step process to do something that it shouldn't do or that I want it to do? Absolutely. And the SQL injection is a great parallel because that is, I feel comfortable saying it's a solved problem in the sense we have prepared statements. I don't. So here is the, we know what the solution is. Not everybody does it. So yeah, you do have to call me out on that one, Tim, fair point. But what I wanted to describe there is, it is still, I think, relatively clear that here was the grammar, the semantics of a query. And here is just the user-supplied input. And that input, no matter what fun syntax it has, it shouldn't change the semantics. It shouldn't have an impact on the grammar. And for those of you playing along, there's nothing like that with prompt injection in natural language for LLMs. And to build on what Tim was saying, even if it's multi-step, it's also multi-content. For example, it might just be that white text on white background, or it's an image that has text in it that the LLM analyzes and just assumes is a prompt or takes as a prompt rather than just the text or markdown that you're supplying itself. So what I want, without getting too technical, I wanted to add on the difficulty of defending against prompt injection, because I don't think that's what we're also telling every organization out there to be as expert as you can on figuring out whatever magical regex there is to say, well, this looks like a prompt injection because it's asking in French or ROT13 or some fancy trick. So what is the better way to defend against this? Well, and I want to add to that because it's not just about that one LLM, it's also about the other tools that it might hand that off to. So if you embed an instruction in an image, as you were saying, and you know that that agent hands those images off to another tool to parse them or to do something with them, it's potentially that other tool that's vulnerable. That's the indirect aspect, right? And there's trust relationships there, yeah. Yeah, yeah, exactly. And so it's more complicated and that's part of why seeing all of the interactions is important to be able to detect those, yeah. I was just gonna add, before we talk about how to handle, before I briefly discuss how to handle like the multi-interaction piece of it, I did, like when I was at a previous company, Social Media, we had a lot of these instances where folks were like providing their phone numbers within the images. Like so, content, not harmful content, but also like engagement bait and stuff like that. And so, we realized that it's not just potentially just an LLM, you can use things like VLMs, Vision Language Models. So, just to kind of explain how folks are tackling that in industry, there are capabilities, there's more AI to figure out the problems with existing AI, if you will, and or just existing threats. But yeah, then kind of pivoting over to the multi-turn, there's several tools that I guess AWS has, but, and then you essentially kind of pipe them together, like all the events that you're seeing. So, you have bedrock events or API calls that goes to GuardDuty. And then there's actually GuardDuty AI where you can actually, if there was like a prompt injection, if there's gonna be a downstream classifier that says, oh, this we think is prompt injection, those events are all gonna be funneled to GuardDuty. And then there's the anomaly detection piece of that. So, we have that capability, but now, again, that's still like single turn. What we're seeing now, and there's some research going on at AWS and obviously at Wallarm based off of some of the tools that they've built, which we'll get into. But, what we're seeing is, yeah, we have to first take the individual interactions, we have to string them together. And then we have to see, okay, do we have visibility into any external tools that are being called or whatnot? And then that gives you a common operating picture of what's going on. It might not be 100%, right? But this is the best we can do. And then based off of that, you essentially then have to summarize, maybe using an LLM or just the traditional machine learning model. And then you ask yourself against a bunch of different dimensions. Oh, do we think this is trying to extract privacy, private information, excuse me, PII, or is this trying to do like a denial of service? Like it's encouraging the agent to call these tools repeatedly. So those different classifiers, and then that information is how you're gonna do it. Now, there's research going on. I'm also very curious. And I mean, I have an idea of what Wallarm is doing, but over to Tim, discuss how you guys are kind of thinking about this problem because I think it is very relevant. Yeah, absolutely. So, from Wallarm's perspective, we have a tool called AI hypervisor that sits inside of Kubernetes and sees all of that interaction that flows through Kubernetes. So we're building what we would call traces, which is the prompt, any tool calls, MCP calls, all of those things I mentioned before, APIs, data, and putting them together into a trace that's actually then associated back to that initial user who started with the prompt. So not only can you see each of the steps and apply policy at each of those steps, detect whether or not there's sensitive data flowing through them. You can also associate that back to the original user who started the interaction. And then detecting anything flowing out of those Kubernetes clusters as well. So if, for example, there's a call out to an external LLM that contains PII or sensitive data, identifying that, giving you the ability to detect or block it if you want to. And then the other side of that is the sort of proactive behavior constraints or policy management where you can use, there's an emerging standard called A2AS. A2AS.org is the standard. You can build these behavior certificates that get applied to agents. And so in our case, you build them from observed behavior. And then you say, okay, great, I now know how this agent is supposed to work. Now constrain it to what it's done based on that behavior. And that's intended to prevent some of the rogue behavior that we've talked about, limiting it to the MCP servers and tools and resources and APIs that it's supposed to be using as opposed to letting it sort of find new ones. Yeah, and very much that's to the point of, we don't want to be surprised by unexpected activity because these agents by either design or necessity have access to sensitive data within the network. And we don't even need to bring in an external threat actor to be worried about this. One of the things, Tim and Ali, you've both been emphasizing that I think it's worth for me to emphasize as well. Originally, I was kind of framing this as an agent does X and an LLM does Y, but you're both very importantly pointing out it might be one human that fans out to multiple agents that also have different varying levels of privileges, data access, read access here, but write access here. That's dangerous. And one of the things I haven't heard so far, and I think I know the answer to this, is how much of the human in the loop should be engaged or I'll possibly rephrase that, could be engaged so it's still a successful security control that just doesn't burn somebody out within one week of trying to click, yes, yes, yes, no, yes, et cetera. I do wanna talk about that human in the loop question because I think it's important, but I think there's a topic there that we've walked around but haven't quite gotten a hold of yet, which is who is the attacker in this case? Because we've talked about the agents as sort of autonomous agents, as the case may be, behaving on their own, taking initiative, but they're triggered from some kind of a prompt. And we didn't really get to how is the attacker landscape changing and what does that mean for the security controls that need to be in place? And the hugging face open AI incident that we talked about to start with, we treat that as if it's an attack, but it's really an incident without a malicious attacker involved. And there are definitely malicious attackers out there who are using AI, and that changes the landscape as well. And that's true, because you pointed out being able to having the traces that Wallarm can track back to who initiated this, and in this case, who the person, which is a great sense of accountability. So I wonder there to build off what you just said, what does it look like? What do you need in place? Or what are the extra concerns that an org has to be aware of to figure out, oh, this is Mike who launched something and it just went rogue, so to speak, or this is evil Mike, who's an external actor and we need to distinguish between these two. I'm gonna add evil Mike to our demo. That's gonna be the name of the attack. I appreciate that, thank you. Yeah, if I may interject there. So we have what's called agent core link tracing. So you kind of look at all the conversations, you start to see what's going on. And just kind of piggybacking off of what Tim was saying, you have identity management, and traditionally it's been, okay, we know who does what. One other critical part of this problem, so maybe I'm not gonna provide a solution. I mean, the solution is more automation, and I can also talk about how security folks need to be in the loop. But one other aspect of this problem is that assuming identities, so you know how you can assume a privilege or assume an identity roles, right? That actually is very dynamic. And so it's like over time, this is what the world looked like. And this agent, they had this ability and they had this privilege, and then they had these intentions, whether that was malicious or not. And maybe at that time, they were prompted by a human being, right? As you march a long time, it's like, well, that can change completely where the agent was just prompted by another agent. And sometimes actually it's like, well, we're kind of thinking a human at some point had to start this conversation. I actually think that that's actually, we're seeing when you have a cron job or something that you run in an automated loop, right? And you just have it running in the background, maybe via EventBridge, another AWS technology, or if you're using GCP, whatever, something that kicks off this automated process, because we do, and I have these systems. So these AI systems, they just start up by themselves, right? And this goes back to the observability, or not even observability, like inventory, right? Like we need to know, we need to discover who's doing what with whatever AI tooling that's available within the company. But yeah, so there is no initial start point. And yeah, so that just kind of makes the problem more difficult. But yeah, now shifting over to the solution there, I think is more automation, but to Mike's original question is, how do we have humans in the loop? And I can share a little bit about what we're doing on some of the projects I've worked on. We, the agent itself will have outputs. So we do have custom annotation tools. So the yes, no thing that you had mentioned, I thought that was very relevant. How do you make that yes, no thing easier, right? How do you make it more seamless for the users? We initially had these annotation sessions to look at one example, whatever that might be in whatever context. It took the security engineers like an hour, like literally an hour. Folks were like then starting to check their phones, you just kind of lose interest and not all the time, but the point being is like, we need to streamline this and make it more efficient. So what we do is we have applications to secure your AI app. You can, in fact, this is one of the best practices that's emerging, at least in my opinion, is that you have the system running. You have it run in the shadow environment always before you actually push it out to prod. You can step out some tooling and whatnot. And then first off, you wanna log the interactions. So going back to that link traceability, okay, did this conversation flow the way I want it to? Did this tool execute after this tool or was it executed before? Why, right? So that's just the, you could say the flow part of it. So that's at just the, you could say like. But then the other side of it is, okay, well, this was the input. At each step of these interactions, the input-output, the annotator, the security expert can say, okay, yeah, this is good. Oh, no, no, this is possibly data leakage, right? And yeah, streamlining that, and then also possibly even pre-labeling it on the annotator tool. We've also explored that. But yeah, this is all, you run it in shadow mode, and then you basically also have the outputs of the system. You basically label it, right? You can say whether you agree or disagree from a security perspective. After those checks, it's kind of like an operational readiness document for AI, right? And so then you make that production launch. So there's a lot of processes. That's just a few, but just felt like that was good to share with the audience. Yeah. And Ali and I were talking separately before this webinar about that operational readiness document as potentially an input into those behavior certificates. So if you know what, because the big challenge with the behavior certificate is you have to know what the agent should be doing in order to enforce that. And that can be a challenge for a lot of organizations to establish. Yeah, even just data labeling, is this sensitive data or not? Orgs are already struggling with that, pre-LLMs, pre-AI. And that's where I did want to, Tim, ask you, because a lot of the story here is, it sounds like the subtext is, Wallarm is able to scale that, it's able to instrument more effectively. So at least if you do know all your AI inventory, I'll say is a little bit of an abstraction, you can at least have it instrumented. So you're collecting the data. Now, I'm curious to how much, based on what Ali was describing, how much do humans have to be part of that labeling or inspection, or how much are you making their life easier? Yeah, I mean, so Wallarm's whole history is around real-time detection and blocking of attacks, whether it's web attacks or API attacks or AI attacks or interactions. That's what we're built around architecturally. So for us, that's the key point. And the human in the loop at this point is, for us, is really before the loop, if you think about it that way. So we want human beings to set up, what do you want to block? What do you want to monitor? What do you want to let through? There's your policy, yeah. What's the policy? And then apply that in real time. So it's not collect data and detect things after the fact, it's block things as they happen. Personally, I think that the future is to get humans out of the loop entirely. I don't think we're there yet. I think it will take us some time to get there. You see that more, there's somehow more forward progress there in sort of the SOC area with sort of agents acting as analysts. But I think the rest of cybersecurity moves in that direction as well. You'll have agentic operators. I think we're certainly, I think that's a key component for any organization because human beings can't keep up with the speed of the attacks that are happening already. And AI is only increasing that velocity. Yeah, and part of, I a little bit glossed over when the two of you were talking about that, like operational doc, the audience for that doc sounded like it was very much the LLMs in terms of, this is what your expected behavior is or is what you should be doing or here are either good prompts, styling, et cetera. Obviously, humans can still read it, but if we can give a LLM some markdown that adjusts its behavior or at least orients it in a good way, it sounds like that's what you were going to. Am I misunderstanding or is this a- I have to interject there because I have a specific story to tell there. Oh no. It's a short story. Where I can't even remember what the task was, but I had an LLM, I had created a prompt for it and I wanted it to be a scheduled agent that would do the thing on a schedule. And I asked it to do its own assessment of the risks. And it pointed out that the number one risk is that it would ignore my instructions and do whatever it wanted. It literally said that. The number one risk is that I'm just not going to do what you told me to do. Fair enough. I don't know. When you take a policy and you hand it to the LLM and say, follow this policy, I'm not sure how effective that is. Is it an LLM or is it a cat? Who knows? Yeah, exactly. Yeah, I think, well, kind of segue. I mean, so like, I think what I was originally talking about with the ORR was, you know, before the application launches, right? You have these annotation tools. And then what Tim was talking about is after you launch, yeah, machine speed, right? You have to have LLMs there. Then I think going back to what you were talking about, Mike, is like you have, you know, this ORR document. How do we make use of it, right? Historically, you have a bunch of software engineers, scientists, sec engines, like security experts being like, okay, we're good to go for production. But to your point, then, you know, we have to then possibly provide this information and additional information for that machine speed monitoring, right? In other production. You're essentially talking about a shift left approach, right? The building of the ORR is sort of a shift left. Let's eliminate risk prior to deployment. You know, and then... Yeah, I think it's kind of like, I think it's kind of like the bridge between the left and the shift right, if you will. If used, yeah, sure. Yeah, because once you have the ORR, that was like all the pre-work you did. You had the annotations. You had all this, okay, this system is gonna behave this way based off of our offline testing, right? And then we give that over to the right. And, you know, on the right side, then folks can actually take that information, translate that into, okay, how are we gonna build a system that monitors our production system? And how are we gonna transform this ORR document into something that LLMs can consume and then behave like basically the analysts, the stuff that you were saying, how security analysts, that role, obviously we're not talking about replacement, but more like how can it be augmented with LLMs? We take that information, translate that ORR document to these agents, and then these agents behave in a reproducible and in a secure way as they monitor the production app, right? So the whole design behind the monitoring tooling or monitor tooling, if you will. So it's like a bridge actually. So at least that's how I kind of think about it. Yeah, that makes sense. Yeah, and it's a good sequence because of like proactive defensive steps, design steps, policy steps, this thing, here's how things should be. And then the reactive steps to confirm, oops, it's not how we expected, just even as Tim pointed out, I would say our policy should be use prepared statements, no SQL injection, but we could look at the number of CVEs this year in 2026. We're getting onto 30 years later, we can still find some SQL injection. And that's where I do want to tease out how much of these, especially perhaps the policy aspect can be either preventative in the sense of technical controls, guardrails, or at least preventative in terms of we're aware of what the threats are or even who or where the threat actors might be coming from. Does that feel like a way that you approach this problem or is there a different way that you would recommend orgs think about securing their swarms of agents? Well, I want to set a little, maybe just a little context on that question with sort of three statistics because if we're talking about what you should do, I think we have to talk about where organizations are today. And so these are from, I think, two sources basically. So just to level set, you know, in this McKinsey survey from last year, 88% of organizations said they now use AI in at least one business function. So, you know, AI is mainstream in terms of usage. I'd love to actually be able to break that down into exact types of usage, but that's fine. 30% said they have reached meaningful maturity in AI strategy, governance, and agentic controls. So by and large, organizations don't feel like they have governance over the AI that they've implemented. And then this last one really surprised me. 80% have already experienced data incidents involving generative AI. Now that's very broad, but it does speak to sort of what organizations are looking at today in terms of the baseline from which they then need to grow. So if we talk about, you know, your question of how should we think about what organizations should be doing? Is it policy? Is it, you know, is it left or right or both? Maybe is one way to put it. The baseline is here, right? There's very real risk and there's activity happening now. And organizations don't have the capabilities or aren't sure what to do about it. Yeah, and I think too, as we continue that conversation, I'll throw back to you in a second, but to also set additional context, we're not just talking about developers using AI and LLMs for coding. Everyone is using it with, and I think that speaks to those numbers you were just showing us too. So there's a lot of this policy and a lot of this observability has to be just more than here's our CICD pipeline. So I do wonder how much of then, maybe going back to you, Tim, what's the first step? Like, an org is struggling, they're in that 30% gap, help us, what do you say? There's a, I think there's a pretty clear pattern here to follow, you know, that occurs every time a new technology, you know, sort of explodes. You could go back to, you know, cloud, you go back to virtualization, to containers, and now we're talking about AI. You have to start with visibility. So it's that first thing of what do I have? I know people are deploying AI, I know they're using it. What do I have? And that visibility needs to include all of those connected components, right? So it's not just the AI, not the LLMs and the agents, but also those MCP servers that we talked about, the APIs that they use, you need those inventories. And then the second step is observability. What are they doing? So I now know what I have, but what is it actually doing in my environment? That very quickly moves to enforcement, which is where, you know, sort of stopping bad things from happening shows up. And I think there's a priority there that organizations tend to address or tend to use by default that we could actually change. They often start with block the bad things. And that's great. You'll start with prompt injection, jailbreaks and that kind of thing, but you'll very quickly move to, okay, but what are the bad things? And then you're in the realm of defining policy and governance. And so you might as well do those two steps at the same time, right? Define what the bad things are and go forth and try and implement controls that block them. And I would emphasize that, you know, we're talking about AI security, but a lot of what we're talking about is not AI security. It's, yes, you have to think about the LLMs and the agents and their behavior, but you also have to think about some of the traditional security controls, identity as an example, API security as an example. One of the things to help manage AI security in your environment would be to like lock down overly permissive, you know, roles so that an attacker can't use those through an agent. And another one would be to make sure that you're, as I said, your APIs are protected so that an attacker can't use an agent to exploit those APIs. So it's a little bit disingenuous of us to focus so much on the AI aspect when it's a bit more expansive than that. Yeah, and especially, I think Ali, that speaks to your environment of AWS. You mentioned Bedrock, of course, that's focused perhaps more on the AI, but the cloud, AWS has so many controls for identity, for network constraints, sandboxing, network isolation, cloud trail for just seeing what's going on. Tell us a little bit about what that, what Tim was just describing from your perspective. Yeah, yeah, absolutely. So, man, that was, yeah, very insightful. Cause yeah, it is, you know, observability, then you go into, well, you have discovery, then you observe, okay. And then from there you have the enforcement, the governance side and within observability, like how do we, like, given we know the inventory of things, hopefully, and that's tooling MCP servers, what exactly, what types of tools do we have at AWS to help with the observability piece of it as it relates to AI, but then back to Tim's point, as it relates to just traditional, you know, application software, right? So we have, as you mentioned, cloud trail, we have Massey. So I know we talked about PII earlier or Macy. It's a system that allows you enable it and it can look at your S3 and your data to just see if you have PII sitting there unencrypted, right? And then you have, as mentioned with cloud trail, you have things like IAM rules, users, timestamps, source IP addresses, when modeling locations happened, right? So API calls, all of that information then flows into like guard duty, which will monitor this information for anomalies. And likewise, Bedrock has invocation logging. So now you're getting into like, okay, so we just talked about some of the traditional software stuff, but then you have the AI events and things going on in just with the AI tooling. So you can enable these model invocations. You can also, like with guardrails, I believe you can say, it's either guardrails or just overall in the family of products that Bedrock has, you can tell it to send information to guard duty via guard duty AI. So it can know how many prompt injections are happening and whatnot, and basically see if there's any patterns with that. So yeah, like this is where like the observability layer of like what's actually going on with your assets, hardware, software assets, and then AI assets, like the tools and the MTP servers. So after you're done with that discovery phase in the observability space, So after you're done with that discovery phase in the observability, there are several tools that AWS has. One thing, as mentioned earlier, looking at, okay, observing conversations, not just the single turn, because we can do that right now, as mentioned with some of the prompt injection stuff and the tooling that we have, but then really shifting over to multi-turn conversations and then how these agents talk to each other. And I think that's where AWS technology, we're building stuff on our own, but also how that complements Wallarm's technology. So I think this is a good segue into what Tim and his team's building over at Wallarm. Yeah, and you try and create slides ahead of a conversation like this that will fit the conversation organically. And then as you have that conversation, you realize, oh, maybe there's a better slide I could have created for that. So one of the things that we're touching on here that I didn't necessarily plan for is the layered approach to securing your environment and securing your environment in AWS. And so we talked a lot about that AI layer where AWS has native security controls that certainly apply to things like Bedrock and Agent Core, and Wallarm has the ability to build those traces and see the whole conversation, whether it's AWS components or not, and then enforce inline enforcement on those traces and those conversations. That's at that AI layer. But at the other layers, there's applicability too, which I think Ali was touching on. At the infrastructure layer, you've got all of that great signal flowing into Security Hub, and there's some powerful capabilities there. Wallarm can also ingest the Security Hub findings and plot them on a graph across all of your accounts, and then do some attack path analysis so that you can see things like where an overly permissive role might allow an attacker to move laterally. That's not AI-centric, but it is relevant when you're talking about agentic or AI attackers. And then Wallarm can also sit at that API layer and detect and block attacks against APIs, which is actually a great combination with AWS's WAF. When you put the two you've got really powerful coverage of web-based web application attacks and volumetric DDoS, and then really API-centric attacks that are focused on API protocols like GraphQL and gRPC. And those are the tools that some of those AI agents may be using in the environment and attackers may exploit. So there's a connection across the layers that I think, frankly, this slide misses because I didn't know we were going to take the conversation in that direction. We would love to continue this conversation with an additional slide that shows that because it's very relevant what you're describing. Even if we just go back to the OpenAI HuggingFace example, HuggingFace saw what, on the order of 17,000 events. That's a matter of volume that they were also just saying they were struggling as humans to recreate, to track down. And I think what Tim is also describing here too is that you don't just want the one point where here is the one agent. What has this one agent, how has it been prompted? You want that all of the agents that have been interacting together, the different endpoints where they've been touching either a tool call or a data access, that S3 bucket, for example. That's where it becomes a much more compelling story to answer that question of, well, what is going on here? So even if you feel some reticence about that particular slide, I think you added wonderful context that really built a story, at least in my head for sure, about what that aid, what that security comes from. I was just going to add, actually, I know Tim and I were talking prior to this. This is very much relevant, just in some way, to the OSI model and how we have visibility into each layer, physical all the way up to the application layer. And what we're seeing is, before AI became a thing, just going back to fundamentals, like, OK, we need to know what's going on at the network level, sometimes even at the physical layer level, depending on your application. If you're Uber, for example, you need to understand your cell phone connection, like Wi-Fi, 4G, 5G, stuff like that. But it all depends on the use case. And then you understand your network. You go all the way up to your application layer. You need to have an understanding of all of that and have monitoring at each layer. And so those traditional tools are just as important. Actually, I didn't know that you could take some of that information that we have from AWS, feed that in to Wallarm's tooling, and then they can do some additional anomaly detection, in addition to what we have. That's pretty cool. And then you can actually work with the web application firewall. So that's pretty cool. Thanks for sharing that. But yeah, overall, it's like, OK, you have these seven layers. And then I think, from my limited understanding, I think it's the eighth layer is like the human layer. But I think pretty soon somebody is going to come out, hey, we need to revisit the OSI model all the stuff. These existing layers, they're correct. But there's also possibly an AI layer, which is really kind of within the application layer and understanding how you add on to the picture there. So yeah, it's defense in depth and having the traditional tools and then these additional tools all in that observability space so that then when you go to the next side of it, which is the enforcement side of things, governance, what have you, you have a better idea of what's going on. It's always about getting more information. And then, oh, well, this is a problem. I didn't know it was a problem, but I'm glad we have the system running that gives us this observability, this additional insight. You're going to get all these marketing people trying to add a layer to the OSI model now. Here's the budget, politics. Yeah, there's all kinds of layers we can throw on here. But to your point, too, you said more information. And I'll just add more quality information, because to your point, yeah, you want like net flows can be very informative, for example, just to not a dig at OpenAI. But as a comment, why is my artifactory making egress to random IP addresses that aren't a PyPi repo or RubyGems repo or NPM? That seems suspicious. And you just need the endpoints, the net flow. But what Tim has been also describing is that you can't just have the net flow between this agent, this agent, this agent, this other agent, this agent, this endpoint. You need to know, well, what's actually the behavior there? So you actually need some insight into what they're actually doing to have that comfort level of, yep, this bunch of agents that Mike, who will be accountable for it ultimately, what he launched are doing the right thing. Or from what Tim has seen from his real-time analysis, we got to go and see like, did you prompt us this way, Mike? Or did you download a skill and that skill actually has something that you shouldn't have been running? And I'm curious, you know, Tim, yeah, how would you, yeah. Well, those behavior patterns, so many of the security tools that we have today, which are built on detecting anomalies, you know, against patterns are built around, you know, patterns that are established by the tools and people that have been around, right? So, you know, we expect, you know, traffic or interactions to behave in a certain way because we presume the, whether we recognize or not, we assume there's a human being behind them, or we assume there's a certain type of deterministic automation behind them. AI agents are going to change that pattern, change those patterns. And we think they're going to be smart. Like we like to talk about how AI agents are going to be sophisticated attackers. I'm not sure it's true. I think they're going to be dumb in useful ways, meaning you're going to see patterns where a human, you know, if a human were iterating through a certain type of attack, you know, they would give up because it's going to take them too long. They might change what they're doing because it's not feasible to do it at scale. The agent doesn't have the same constraints. And so you're going to see very interesting and different patterns that we're going to have to adapt to in terms of detection. And so, you know, the agent as attacker becomes a very, you know, interesting topic to pay attention to going forward. Oh, sorry, go ahead. Yeah, go ahead, Ali. Oh, I was just going to briefly add something to mention, like, this is tangentially related, but there's, you know, physicists and mathematicians that are doing research using LLMs and AI. And they've observed the same thing. It's like the LLM can just keep ruminating and thinking about something. It can explore all possible paths, whereas a human being, we think, oh, we have to just try this complicated approach to solve this problem because that's really how much bandwidth we have. We have finite time, finite resources. What we're seeing is like, oh, well, simpler approaches can actually solve the problem, but we would have never have thought of exploring that. And so there are some evidence that, like, even in just research to solve some problems and get further ahead, it's, yeah, the ability for the AI system to try all possible things and whatnot, so. Yeah, you might have a denial of budget as it burns through tokens to do that. But to Tim's point, you still have that brute forcing to a degree, just trying and trying and trying. And they're perhaps not limited like the time that we're limited for webcasts like this. But as we start to wrap up, there's a little bit of, I want to play with a little bit of history of InfoSec, if you will, to riff off SQL injection, for example, that's 20 plus years old. But we also could have a policy that says, use prepared statements. To S3 buckets, for example, early on in the early days of AWS, sorry, Ali, it was very classic to have, oh, here's a sensitive data in an S3 bucket that was open to the internet. Now, by default, the policies are S3 buckets are private. So in other words, InfoSec is learning. Can we compress the decades of that learning for SQL injection and S3 buckets and apply it to AI? And if we did, do both of you have a policy or two that you would love to snap your fingers, and that's enforcing security for AI these days? It's a good question. I mean, I think yes, we can compress the time frame. And we already are, frankly, because that's good. You know, it just doesn't feel like it because AI is also moving, AI adoption is moving faster than previous technology, you know, shifts. And security is still behind, but it's not further behind than it was with previous technology shifts, if that makes sense. It does. And I appreciate the optimism. Yeah, I mean, I think there are lessons being learned there actively that compress the time frame for security to sort of be meaningfully useful. But there are some things we can't learn until we learn them, you know, until they happen. That's certainly true as well. The question about what's one policy that you would put in place, that's a harder one for me. Because I'm not sure that I can come up with just one. I do think that the biggest challenge that organizations should address is understanding the agent's behavior. And I'm thinking specifically of production agents that have been deployed, not so much the development agent that's writing code. There are challenges there. But I'm thinking about the agent that interacts with your customers either directly or indirectly and takes actions on their behalf. That's where the behavior drift is a problem. I mean, I do think that we're in this position of expecting deterministic outcomes from a system that is intentionally designed to be non-deterministic. We want an agent to take these actions and behave the same way every time when LLMs are specifically designed not to do that. And so maybe that understanding is sort of the key point I'm trying to make. Yeah, focus where the negative impact could be the biggest. I think that's a great recommendation. Oli, you want to take us out with some final bit of wisdom from this last hour of conversation? Yeah, absolutely. I learned a lot. Thank you for everybody's time. And this was a great chat. I think one policy I can think of, and to Tim's point, there's several, but one thing that I think is like a practical insight that we've seen is by default, tools. So you should first classify whether the tool has network access and then what types of resources it's hitting, what types of repositories, whether that's a token repository or whatever. And then basically a default policy of if you have network access, it should be off. And then essentially the AI is going to fail at what it's doing. And those failures are actually good for us because it's like, okay, it failed. All right, should we enable this? Should we not? And yes, to the earlier point of the conversation that eventually we might have an automated system that determines, okay, should this get access or not? But this policy by default is essentially tools that are risky or have the potential to be exploited. They're by default turned off. And that can be implemented on the left side of things, like at the software side of things, before you push something to production, you can have scanning and you can say, oh, wait, you're giving these privileges to this tool. Did you do any kind of analysis? Did you get approvals from a security expert, whatever? And then also at the production side of things, like actually having access control mechanisms in place. So that is one policy I can think of that has served well historically. That's wonderful. And for as much as both of you have been talking about automation, AI, and trying to minimize humans in the loop, I very much appreciated having the two of you in the loop have for this conversation. So thanks so much for this, Tim and Oli. Yeah, thank you, Mike. Thanks for having us. Thank you. Thanks, everyone who joined us. Thanks again to Wallarm for sponsoring this webcast. And if you didn't scan the QR code to find Wallarm on the AWS Marketplace, that will still be available in the resources after this. Promise it's not even a rickroll either. Legit link. So thank you to everyone who joined us today. And thank you to everyone who's listening to this recording in the future. Please do check out wallarm.com and keep an eye on scworld.com for more expert webcasts like today's.
