Transcript
that a lot of times the business doesn't understand. And so building the community, reaching out, listening to stuff like this, listening to other podcasts, but creating the understanding that this is an industry that is dealing with this and the industry is fighting. We're all fighting the same fight against the same people. We're not fighting each other. And so reach out, ask for help. I'm David Moulton, and this is Threat Vector. Today, I'm speaking with Asaf Karan about AI in the wrong hands. Asaf is SVP and Chief Security Officer at Qualtrics and the author of a new book, Lessons from the Frontlines, out now from Wiley. Welcome to Threat Vector. I'm really glad to have you here. I know there had been some scheduling nonsense, but we finally got it, man. We're finally on the mic together. So let's have a good conversation. Six rescheduling to get to this point, if I counted correctly. But let's go. I'm excited. Before we get into our topic, I'd actually like to hear a little bit about your journey. I dug into it a bit, and I'm sure our guests would actually find this interesting. You've had actually a fairly long career in cybersecurity from your early work in Israel, through PayPal, and now with Qualtrics. How do you think about the path that brought you here? There were really hard points in my career. I had to step out of a startup because I had lack of clarity and lack of cohesion with my co-founders after a year. And that was really, really hard. It was probably one of the hardest years in my life, but also probably one of the years with the most learning for me. And what I would say is that I didn't know that at the time. If you look at the different decisions that I've made, I didn't know that at the time, but there was always those search for experience, not for title, that guided my career progression. So when I went to run a startup, it was I wanted to do this thing. I want to try running a company and I want to do that enough twice. I don't want to do that again, by the way. But even leaving my second startup and going to work for PayPal, I took a roll cut. I moved from being a CTO to being a manager of four people in EMEA. Ended up being the CTO in PayPal, which was great decisioning, probably, hindsight, but it was searching experience. It was searching what is the experience gap that I have to make me a more full professional. And that's what I suggest to people when they come talk to me, especially people that say, hey, I want to be a director or I want to be a senior director. I want to be a VP. It's not a good pursuit. The pursuit is I want to do something I enjoy and I want to learn new things. And this is the direction that I want to go to. And I think that in hindsight, that's what's driven my career so far. Yeah. Well, in your book, you wrote about this danger of feeling like you know enough and how that confidence can become quietly become a liability in a field that's moving as fast as AI security. I think that trap feels easy to fall into. Where do you see that showing up now? Specifically with AI, I think that I'm seeing a lot of security teams not understanding how pivotal this moment is and using legacy thinking in making decisions and maybe defaulting to to the default of security teams, which has been the Department of No. I think especially there is a gap of knowledge in security teams understanding AI and machine learning. I think it has been there for a while. But with the explosion happening right now, that fear is dangerous. And I, that lack of curiosity that I'm seeing in a lot of places is bothering me, because I think that we're creating more impact than good when we're doing it. How do you catch yourself from falling into that trap? How do you catch yourself from falling into that trap? Sometimes successful, sometimes I'm not, by the way. I don't want to make it sound like I'm always curious, but I do curiosity checkups. I sit down and generally say to myself, what did I miss? There is a friend of mine, Leah, who's the CISO of LinkedIn, and they wrote on LinkedIn that something I agree with completely, that there is a superpower in willing to look like you don't know the answer, or willing to look like you're stupid and ask questions like you're stupid. And sometimes I'm successful, sometimes I'm not. In the day-to-day, like the accelerated day-to-day pace that we're in, a lot of time it's just easy to come in and say, hey, this is the answer, move on. And I do have a good team around me that knows a good team around me that knows to also challenge me when I'm that way, and tell me, hey, Asaf, you're wrong here, let's have a conversation. And that's really humbling, and it's great to have that support structure. Yeah. I have concluded that there's a difference between being dumb and being stupid. And I think being dumb is acceptable. It is a natural state. All of us are dumb. And when we refuse to learn or refuse to learn the lessons that whatever the situation tries to teach us, that's being stupid. So you touch the stove the first time, sorry, kid, kind of dumb, now you know better. You touch it the third, fourth time, now it's just getting kind of stupid. So I think that it's okay to look dumb. In fact, if you never look dumb, you're not really walking into situations that are going to challenge you. I think it's when you look stupid later on when you have that opportunity to go learn or to understand or to be curious and dig in, that's when we end up looking stupid. And I think that's what we try to avoid, not realizing that I'm grateful when I'm in a room and I'm the dumbest one. It means I'm about to learn some things and maybe have a lot of time to reflect and think and ask good questions. I like the idea that you have a team that's around you that can push you too. I think that's a sign of a strong leader when your team can push back on you and say, we got to rethink this. All right. A quick question for you before we get into the deeper topic. You've had this really long career. Obviously you've got tons of stories to tell. Why this book? Why now? So I've been wanting to write a book just for the experience of writing a book for about five, six years. Okay. And I kept hitting a wall of what do you have to add to literature that wasn't already written? And that's valuable for other people. And I, and it got to the point where like, okay, I'll start writing something and then stop and then start writing something and then stop. And I said, well, you know what? My stories are my stories. And I like to tell them, maybe I write a book based on my stories. So I sat down and wrote a lot of different stories. And I wrote So I sat down and wrote a lot of different stories that I have. And then frame, start framing frameworks in my mind and like curiosity, grit, and optimism, and the diplomacy, business acumen, change management, and execution, like frameworks that I use when I talk to people all the time on how to build things or to build teams or how to behave. And I said, okay, now I have stories. Now I have frameworks. It comes together into a book that is based on my personal experience. And none of this is groundbreaking, but it is, I think, the first time that was written in the sense of a security leadership book and with some grounding in the life-to-life that we deal with, which is a bit different. And the part I'm proud of the most is actually the last part of the book that is a lot about psychological safety and taking care of yourself and acknowledging the mental challenge that is working insecurity that is very different to other roles that other people play. And that came together really, really nicely into a book. And so the actual act of writing the book, that was a few months. It was pretty easy once I had the structure in place, but getting there took me years. I'm hoping that people find it useful. I don't know. We'll see. I recently interviewed Allie Mellon about her new book, Code War, and her big lesson was be aware of time management when she put the book together. And she said she learned a lesson that she didn't have her time management as tight as she wanted. And it sounds like yours was finding that truth that you wanted, and then it flowed out of you. So let's get concrete on this one. I appreciate you letting us go behind the scenes and get to know a little bit about you, but let's talk about what happens when AI gets in the wrong hands and what that actually looks like today. Maybe not like a future scenario, but what are you seeing attackers doing with these tools right now? Great timing. I just published a blog post about the whole Mythos thing. And I said in that blog post, it's not a future conversation. The fire started in 2023 when GPT was unveiled. Initially, what we saw was the basic things, phishing, deep fakes, those kinds of discussions. I think a month ago, Amazon has published that they've seen an attacker go in and do it and call takeover, go in and utilize AI agents to do discovery within the network or within the customer environment. We're definitely seeing condensation of the time frame from vulnerability to execution of the vulnerability. AI red teaming is a real thing. Red teaming is a real thing. And if AI red teaming is a real thing, where AI attacking is also a real thing. And all of these things are reality right now. And the interesting piece about what happened with the Anthropic Mythos publication is that everybody say, oh, this is going to be bad. No, it's bad already. And I think a great example of how this shifts things drastically, I had a conversation in RSA, it was already a month ago, oh my god. I had a conversation in RSA with a founder of an email security company. And I asked him how it's going because they raised a seed and they were going. And usually when you raise a seed, you go to the US market and you start there. And he said, we're actually big in Japan. I told him, why? Why Japan out of all places? He said, look, one, the Japanese culture is very trusting. It's one of the safest places on the planet. You can leave your wallet and on a desk at lunch and nobody will touch it. And so inherently, it's a very trusting culture. And that's great. But they have had a language and a culture moat around phishing all of these years. And now they don't have that anymore because GPT and other models are able to mimic Japanese well enough. So phishing now has become a pandemic. And the government is very focused on that. So we're getting a lot of traction in Japan. And we're going to see a lot of these shifts in which the assumptions that we've made on things that will keep us secure are going to be null and void. And we will need to change the way we think to building better cultures and better systems. And that's the reality right now. Now, will it get worse in the future? Yes, it will get worse in the future as we improve models. I like to say, when I talk to people, attackers don't have security teams telling them not to use AI. Corporates do. And we're the security teams who are making this bad for our companies. But they will continue using it. So when you think about AI-generated phishing, you mentioned that with the Japanese market losing that sort of natural or just that defense they had because there was the language barrier. LLM-assisted recon, deepfakes, right? There's all these different tactics and things that AI is helping an attacker with. Is there a capability that really stands out to you more than the others? I think that we are already seeing semi-autonomous, if not fully autonomous, agentic attacks. And that means that the scale and scope, the economic pressure on people that the attackers have is going to reduce. The scale and scope of what they can try is going to be or already is accelerated. And they're going to get to vulnerable endpoints quicker. So it's about how fast they can move, which is scary because we can say as much as we want to say security by obscurity is not a thing, but security by obscurity is a thing. And unless we take a really strong stance against it, then we're going to be bitten in the ass by these attackers that now don't have people constraints in doing fully autonomous recon. The other thing that I'm worried about is dedicated crafted malware that does not have signatures. Okay. I want to get into both of those, but my first question for you would be, what does that AI-assisted or agentic attack look like at scale? I've tried to look back at some of the big attacks in the past and imagine a world where they weren't human capacity constrained. And it's unsettling for me to think about that. But walk through that for me. I think one is once you're in the crosshair of an attacker, then the enumeration discovery of the endpoints that allow entry into your environment is going to be very fast and very thorough. And then the attempt to hijack those endpoints is going to be very fast and very thorough. Probably noisy at start, which is where we have some level of, if we use AI or if we're fast enough, we're able to find it and catch it before it comes in. But that's going to happen. And like I said, security by obscurity is not going to work anymore. Leaving an endpoint that people don't know about up in the open is just not a good thing anymore. It was never a good thing, but it's even worse now. And then once you go, you have somebody in the network, the speed and ferocity maybe, speed and ferocity of them going and accessing data and taking the data away and exfiltrating is something that we've never seen before. And you have seen like living off the land type of attacks where people were trying to install OpenClaw on devices after they breached them. But OpenClaw that they managed. So getting to persist through AI, which is also very, very interesting. So Asaf, one of the things that you may have noticed, and I certainly have, and it's counterintuitive to think this way, I think, is that there's a lot of focus on AI. And I think that that is warranted. On the other hand, have we pulled so much of our focus away from some of the basics that seem like we need to be able to go in and deal with the discipline and grit work that isn't all that sexy and new, but needs to be done such that the attack that you're talking about isn't so damned easy? Yes. Yes. Thank you for that. We need to say this more. The best solution, two good solutions for AI attacks. One is minimization. If it doesn't need to be on the internet, it shouldn't be on the internet. If it doesn't need to be on the endpoint, it doesn't need to be on the endpoint. If it doesn't need to be in the package, in the source repo, it shouldn't be there. And we have been in a world where we're maximizing things. We need to minimize. We need to reduce the attack surface to a point where the attack is not possible and not get to the point where we're trying to defend a growing attack surface. And the other is baseline boring architecture. We need to do identity right. We need to do data right. We need to do scoping right. We need to do network segmentation right. We need to do recovery, BCP right. And these are hard things. And we've been glossing, as an industry, we've been glossing over them with mitigating controls and good enough and all of those. There is no good enough anymore. Because what we're doing is even worse than attackers using AI. We're putting AI on top of broken mechanisms. And so we're putting a non-deterministic engine on top of a broken deterministic architecture that can go and do whatever it wants. And our ability to control a non-deterministic engine is very, very low right now. Until we get into the world where there is runtime security for the AI solutions that we provide to our customers, there has to be very strong architectural guardrails on the bottom. And if we put on an AI agent on bad identity infrastructure, it will find a way through prompt injection, through other means, through, I don't know, to get to the data that he wants to get to or the attacker wants to get to using our own bad infrastructure. So completely agree with you. There is, in my mind, a whole resurgence of being brilliant at the basics. Yeah. I mean, sometimes this idea of if everyone's going to zig, it's time to zag. And a lot of oxygen is used up worrying about a version of a problem that we see coming. And then we're distracted from the problem that we have, you know, this security debt, technical debt, whatever you want to call it, where that's just sitting there. And I think you've said publicly that when you bring an AI tool into your environment, you have less slack. I think that was what you were just describing. And that you can't skip the steps. And I know you mentioned some of them, but I want to hammer home on this. What steps do most organizations skip? And which one of those exposures do you think is going to end up being the one that haunts organizations the most? Identity. Identity is probably the hardest, especially in product. If you're a SaaS company, or if you're even a consumer company, identity is probably the hardest piece. In a lot of places, identity was homegrown years ago. Customer identity was homegrown years ago. And there are best practices there. And there is not a lot of people that know how to build it right. And if people miss identity, that's the baseline structure for everything else. Years ago, a CISO told me that there are three rails, and the third rail is identity in any CISO's job. And I want to say data was one piece and network was another. But like those were not the ones that if you touched them, the business would zap you. It was identity, because you had three, four different identity systems. Some of them worked for the executive owner. They didn't all work together, and they certainly didn't work well for security. And it seems like now we're at a point where that being the third rail as a mental model for a security leader has to flip around. It has to be the first thing that you're looking at, and getting right, and getting right really quickly, or you remain exposed. The second is data, by the way. Like it's very close second, but identity. I would go after identity first. So I know a lot of security leaders are being asked to make decisions about AI risk faster than guidelines can be issued or updated as things change. What does good judgment actually look like in this environment where the threat intelligence on AI is drafted? Maybe it's being written. It's coming in over the weekend out of a user group who felt the need to put something together. This stuff is not tried, true, tested, public comments are done. It's really fresh. How do you operate in that environment? It's also changing very quickly. It's also changing very quickly on an ongoing basis. So what you've done a week ago can change next week because new model, new capability, new thinking. I think that going back to basic principles is important. What are we trying to solve? Where are we trying to solve it? I think that being realistic about the risk is important and understanding because we as a community, we have a tendency to over-exaggerate risk because we don't understand it because it's changing so fast because it's this new thing and there is hair on fire and people running around when industry is steeped in FUD and we need to figure out how to deal with it. So I think that's a big part of it. And we need to fight it. Also, we need to understand that it's not going away. I know security leaders that in 2023 said, yeah, yeah, this will be a FUD, it will go away. No, no, it's not going away. This is part of the future. We need to lean into it and not the other way around. I think when we try to block the business from using AI, we're creating more risk than value. We need to sit and create mechanisms in place to allow the business to use AI in a secure and reliable manner, knowing that we're taking risks, but we need to enable the business to use AI. We need to build guardrails around that. Now, there isn't a lot of enterprise software that is there yet that is doing all of the things that we need to do. So we're going to need to do a mix of vendors or a mix of internally built stuff and a mix of externally built stuff and open source and stuff like that. But building the guardrails to make us feel good about, or better, not good, about where we are from a risk perspective is important. So in a lot of places, what I'm hearing from peers is that use AI to use AI because the board said use AI, which is a wrong framing for that conversation. You use AI to get to an outcome that is a better outcome with AI. And so I think what we've managed to do internally is say, hey, we want to do these things. We want to automatically triage all of our SOC incidents with AI, or we want to do vulnerability triage with AI, or we want to do questionnaires, customer questionnaires with AI to free up people so that our people can do bigger and better things. Those are really important outcomes. But I don't feel the push on, oh, just use AI for AI's sake, which is something that I'm fortunate about. Yeah. Don't go for the next job title because it's the next job title, right? It doesn't make sense to apply that logic on using AI, especially when it is a tool for an outcome, not the outcome itself. Let's just say that a security leader is listening right now, and they're not sure how exposed their organization really is. Maybe they heard you say that we're over-indexing on the risks, and hopefully that's true for them. But they're trying to figure out where to start. What's the first thing they need to do? So two lenses to this. This is the internal AI exposure, people using AI or products using AI within the constraints of their organization, and then the attackers. When we talk about the internal piece is get an understanding of usage because you're going to trust some vendors, and you're not going to trust other vendors. And this is very, very important. You're going to need to make decisions on which vendors or which hyperscalers or AI vendors you're going to trust, or which SaaS companies you're going to trust and which SaaS companies you're not going to trust, or which hyperscales you're not going to trust because you don't think they have the right controls in place, or they have the right structures in place, or they're responsible enough, not responsible enough. There are a lot of things that you can trust, but you can't trust understanding usage is extremely important, and starting to build guardrails on that usage. And if you're building your own models, if nothing else, you can go do ISO 4201, but if nothing else, look at the NIST AI risk management framework and start looking at how you're going to build your own models. And if you're building your own models, you're going to have to build your own models. And if you're building your own models, look at the NIST AI risk management framework and start looking at how you build your model inventory and how you build your model risk scorecard, which is extremely important, and try to at least publish it internally so that people understand the different risks in using different models, bias, ethics, operational risk, not operational risk. These are baseline things. So that's what I would say for the internal risk. For the external risk, attackers using AI to attack companies, I would ask, where are the places where you can be much faster if you utilize automation? Go and automate. With AI, without AI, I don't care, but go and automate. Where are the places where you can be better if you are reducing attack surface? And you can do it fast. Go and do that. Start building both speed and reduction of the attack surface as soon as you can, because those are the things that are going to save you. The other things are. Yeah. Attack surface diet. Attack surface diet. I like that. I'm going to use that. Yeah. Yeah, you got it. I'm stealing brilliant on the basics from you. Yeah, yeah. Well, I stole it from a guy called Shishi Fernando, who was my boss in PayPal. So let's go. I think he stole it from a guy called Wasamu, who was our head of SRE. I do a weekly post to my extended leadership team. So attack surface diet is going on the next one. I mean, it's good practice, and it's maybe now a required practice. Get your attack surface on a healthy diet. Shrink down, man. Attack surface calorie counting. Yeah. Get that beach body attack surface before summer. Yeah. Summer is coming. Yeah. Yeah, there you go. AI summer is coming quick. I want to end on hopefully a positive note. You've written this book about what it takes to lead in this field long-term. You're watching everything that's going on with AI right now. Is there anything that gives you confidence that defenders may come out ahead in this era? Yeah. To steal a quote from Phil Venables, I'm a short-term pessimist, long-term optimist. I think that the next couple of years are going to be either hilarious or daunting, depending on who you are. But I think in the end, this technology is so exciting that we're going to be able to do something that we've been trying to do for years and years and years unsuccessfully, which is to free up people to do people work and not to do manual labor tasks. We're already at the deficiency of the amount of people in the profession, and people are burning out because they need to handle incidents on a day-by-day basis. So copy-paste answers into questionnaires or do third-party risk management things that don't bring value but are part of the process. We're going to be able to automate a lot of these processes and reduce the amount of time people are actually doing stuff like phone triage or incident triage and have them work on the larger picture that it's going to be much easier. Not easier. It's going to be much more exciting to be a security professional in two years than it is right now because you're going to work on big picture stuff more than you are today. And I think that that's exciting. And I think we will get ahead of the curve. We need to adopt the technology as fast as attackers. That will not happen. So that's why we have two years of catching up. I think we'll catch up in the end. So I've been trying to think about the future and what it might look like, and I found this image of a 1920s potato farm. And there were laborers digging and working in the field. And then I contrasted this with a vertical hydro farm. And they are as far apart worldwide as jobs go. They're both farms. But I do wonder, are we in a moment where we are laboring and digging and trying to keep that potato farm going? And we're going to transform into one where it's a controlled environment. We have incredible productivity, some level of small team being able to handle that vertical farm of the future for security. And I'm hopeful for that. And I look at the potential. But I think that the first thing that has to happen is we think differently about those basics, those fundamentals. We go on that attack surface diet. We put together a different model that allows us to control the environment and flourish rather than try to work harder and longer and not have much effect. So I'm hopeful. And I like to hear that you think that it's going to be two years and we're through it, given the time lately. Maybe it's two. Maybe it's two months. Maybe it's two years. Yeah, we'll see. Fantastic. Asaf, thanks for the great conversation today. I really appreciate you sharing your perspective on AI in the wrong hands, but also letting me get a glimpse of your path, your art, your wife's art. Folks, Asaf's written a new book, Lessons on the Frontline, Insights from a Cybersecurity Career. It's published by Wiley. It's out and available. We'll have a link in the show notes, along with the blog that you mentioned earlier. And I appreciate you coming on Threat Vector today and having this conversation with me. Thank you very much. Appreciate it. It was a lot of fun. That's it for today. If you like what you've heard, please subscribe wherever you listen and leave us a review on Apple Podcasts or Spotify. Those reviews and your feedback really do help me understand what you want to hear about. If you want to reach out to me directly about the show, email me at threatvector at paloaltonetworks.com. I want to thank our executive producer, Michael Heller. Original mix and music by Elliot Peltzman. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now.
