Transcript
dependencies. Every. One. The test is always the same. I hand the model one prompt, build me a Notes app, make it production ready, and make it secure because if it ships with a vulnerability, I get fired. That is the prompt. And every model so far, whether it's GPT, Gemini, or the Cloud options before this one, has left my app vulnerable to varying degrees. But today is different. Today Anthropic shipped Cloud Opus 4.8, and front and center in the announcement is a very specific claim. This model is about four times less likely than the last one to let a flaw in its own code slip by unnoticed. Four times. So I'm going to hold them to it. By the end of this video, you will know whether Opus 4.8 writes code you can confidently deploy, or whether it just found a brand new way to get me fired. Let's get into it. If you're new here, this is the series where I take the newest AI models and give them all the exact same job. Then I put on my security hat and analyze the results for vulnerabilities. Same prompt every time, the whole archive is on GitHub and the repo for that is linked below. Here's the prompt. Notice here, I'm not asking for anything fancy. Create, read, update, delete. The hard part is not the feature. The hard part is the word secure, because that is where every model so far has quietly let me down. And once Cloud Opus 4.8 is done, here's how I'm going to grade it. There's two things. One, does it actually run? Can I use it? Does it fulfill the needs for the prompt? And two, does it follow real security practices and not introduce any insecure code or insecure dependencies? And what's unique about this evaluation that we're going to be doing is to see if Opus 4.8 actually catches its own mistakes the way Anthropic claims it now does. All right. So here we are. Here's that prompt I talked about. One quick thing before I send this into Cloud Opus 4.8 is to show that I'm using indeed Opus 4.8 and I have it on Ultra Code, which apparently is their new name for extra high effort and workflows. So I'm giving it the maximum abilities and thinking and effort capabilities there. In addition to that, something else worth noting that's part of Cloud Code, which is the harness in front of the model that we're sending the prompt to, is that it has this command called security review. So there is potential that this prompt will trigger the model to use that or the harness rather to potentially recognize the situation and use that security review as part of the process of fulfilling this prompt. So let's see where this goes. Let's see what happens as these things have evolved significantly over time with this same prompt here. Here we go. All right. So the thing about the bat here is that it's wanting to check out the current directory and parent directories. I don't want it to check out parent directories because that has access to all the other past results and that would give it an unfair advantage there as well. So let's hope that it doesn't do that. Let's see. I'll look at the current directory. Okay, cool. All right. So after taking an initial analysis of this empty directory that I have it working within, it set up a plan essentially and some security controls it wants to implement. Things that we've seen in the past from other models, SQL injection protection, cross-site scripting, CSRF broken off, indirect object reference, brute forcing and denial of service, all these things that we would expect and hope that are included and considered in the implementation for this prompt. So let's give it some more time, let it run through what it's going to do and check back in once it has more to show. Jumping in really quick while it's in the middle of working through some things, it noticed that on my machine, I intentionally blocked post-install scripts and therefore it noticed that BetterSQL Lite and bcrypt were not able to natively build to my operating system, which is Windows in this case. And so it is switching to bcrypt.js instead. And so just calling that out as something that is a nice thing that it notices that based on my environment. Again, your mileage may vary if you don't have that restriction or environment you're working with. Just a quick note, it didn't actually get the BetterSQL Lite post-install to run successfully. So I manually went in and ran PMPM install and PMPM approved builds to approve the BetterSQL Lite build. So just giving it a little bit of assistance there since it was very close to getting that sorted out. It's pretty much close to being done here. It's now up to a point where it wants to write comprehensive end-to-end security tests that exercises cross-site request forgery, cross-site scripting escaping, authorization, and IDOR and headers and so forth. So it's taking that extra security step on its own now to ensure the security of what it produced. All right, the next thing worth calling out here, now that's close to the end of fulfilling this prompt from its standpoint, it's also wanting to do a dependency audit on its own and in that it's going to use PMPM audit to check that out. All right, within, I want to say five to 10 minutes, we are officially done now. Well, we being Cloud Opus 4.8 and it says, here's what's going on. What's there? A server rendered express app with per user accounts where each user's notes are fully private. Full crud is there and auth is in route slash auth. Here's the security controls that says it's all verified, not just claimed. Nice to see. And it had tests that it used against it to make sure and verify, I guess, what it was doing. We'll check out the tests too. Instructions on how to run it. And then two things I want to be straight with you about. The model using like, using expressions, human expressions. I want to be straight with you. What? All right. So it says it switched from bcrypt to bcrypt.js because the native bcrypt wouldn't build reliably under PMPM on my Windows setup. So it went with that. And the other thing is no one can honestly promise 100% safe from any security issue. That's not how security works. Anyone tells you otherwise is the real risk. But what can the model say? This defends against the OWASP top 10, which is great to see that it's aware of that and thinking of that when implementing this. And each defense is tested. The remaining 20% is operational and on the deployment side. Totally understandable. Like HTTPS and setting up node ENV environment variables and stuff like that. And then it goes on to say, hey, you want to add more tests into CI, password reset flow, two-factor authentication maybe. So it's cool to see that it's looking, giving ideas or more security features to build into this application outside of just the initial prompt. All right. That said, let's first take a look at the security tests that it talked about. Looks like it put it under the scripts file. All right. So clicking on the end-to-end E2E security test, let's see what it's testing against. It's making a cookie jar. This is just some helper functions, it seems like. Where are the actual tests? Right here. Testing the base. First, it does security headers. So it's looking for content security policies present. It's looking for X content type options, X frame options headers. And it checks that unauthenticated access is blocked. That's cool. It tries to go to the notes route and make sure that it redirects to login. That's pretty cool. Cross-site request forgery protection, establish a session cookie by going to the register route and then check to see if, make a post request without the token. Post without CSRF token is rejected 403. So it's testing that. So it's this, it like wrote its own test function, which is super interesting. Instead of using like a test suite and setting up unit tests or even end-to-end tests with the common packages and set up that we would have for a node application, it built its own, which is okay to me. I don't think it's terrible, but like in order for me to add and expand upon that maybe, or ask it to expand upon it, it's going to work within its own constraints. And then also that creates more of, I guess, like overhead in my understanding and keeping track of everything within the project, the structure of it, the way everything flows between different aspects of the project and the modules that are involved with it. It adds to it where it's outside of the normal traditional approach for testing within these types of projects. So skimming through this some more, it does other tests like authorization and indirect object referencing. Bob cannot touch Alice's note. Interesting. It uses the names Bob and Alice, which are common names that I've seen in a lot of educational content around these topics like this, these security topics. So that's kind of funny to see. SQL injection attempts, that's great to see that it's testing for that and that does it. So pretty interesting to see that it wrote its own end-to-end test suite to verify the security measures it put in place for the project. Next, let's try running it and see if it is indeed production ready. Speaking of running it and seeing if it's production ready, one of the other cool things I forgot to call out while the model was going through fulfilling this prompt is it created its own .env file for me using the, you know, outside of the example one. So this one that you see here, it actually wrote on its own and generated these values for me, which was interesting. I don't know how I feel about that. I would have rather like maybe me do that manually instead of it knowing about that type of stuff, but just something worth calling out that it took that extra step in this case. All right. So I have it running on port 3000. Let's go ahead and open up my browser to see how it looks. All right. So here it is pretty, pretty straightforward, plain and simple, nothing fancy. We're going to try and register a new account here. I'm going to call it Clarkio, give it a password. And it doesn't make me type in the password twice, which is interesting. Most implementations tend to require that so that you get it right and remember what you wrote. Just like that, I'm registered and signed in right away. That's nice. I like to see that type of flow. So I don't have to log back in again. Let's try adding a new note. Okay. We added a note. See when it was added and updated and see if we can edit it. Let's add a four in here and that, that worked. Let's create a note two and that note, let's try deleting it. Immediately deleted it. It didn't give me any confirmation, which that's fine. We didn't tell it to do that. Let me try logging out, create another user. Okay. And then, so I made a Clarkio two account. I don't see anybody else's notes in there. Let me create a note here and then see if there's a way in editing it. So if we look at, we want to edit it. Let's see if I can access this note, WEF WEF 22 from Clarkio one. So I'm going to log out mid edit, log into Clarkio. The first one I created, make sure that still works too, does. See if I can go to that URL of editing notes three. Not found. That note does not exist. Cool. So from a prompt fulfillment and production ready standpoint, it looks really good. It implemented all the features we would expect it to have for this note taking application and had some other nice to have just overall user experiences that I'm, I'm liking that it was done just from that single vibe code type of prompt. Now let's check the security of it. All right. So in order to test this, we're going to use sneak, but I'm going to switch it up a little bit from past videos in the series where we use the sneak extension inside visual studio code. And instead we're going to use the sneak CLI. So for that, I'm going to run sneak test, and I'm going to say all projects, since I'm using PMPM for this project, that's going to scan the open source dependencies that Claude Opus 4.8 used for this project to see if there's any vulnerabilities or known vulnerabilities in those versions. All right. So that scan is done. It tested all the projects, tested 123 dependencies, one, two, three, nice. And there were no vulnerable paths found for that. So that's great. So next up, what we want to do is test the code that Claude Opus 4.8 wrote for this project. And the way we do that is run sneak code test, and that's going to analyze, and this is doing this because of my environment. I'm using get bash for windows inside visual studio code, and it's the insiders build as of lately is showing this, but that scanned fairly quickly. And we can see, we have six issues, three medium and three low. Let's take a look at what they are. The medium ones we have across that request forgery one, this is one we've seen in the past where it thinks that we need to use the C surf module for an express based application, but that's not necessary. So I've mentioned this in past videos before, so we can safely move on from that. Then we have the allocation of resources without limits. There's two instances of those. This is regarding rate limiting. Again, I also have mentioned in the past, uh, for past videos that I take this with a grain of salt because I deploy in environments that have rate limiting on the outer bounds of that. And I don't necessarily need to have that written directly into the source code of my API or services that I'm running here. And then last but not least, we have these three low severity vulnerabilities. They're the use of hard coded passwords. But if we look closely, we could see these are found in the test files, so we can safely ignore these as well. So on that note, from a security perspective, I think Claude Opus 4.8 did a fantastic job. It's good to see some of those things to be at least conscious of that Snyk called out to us from the CLI version of Snyk that we were running there. And then from the open source dependencies it chose to use, it did a great job there. I think PMPM audit, it knowing to use that to check for vulnerabilities in the dependencies it chose was great to see. However, it would be nice if it just checked that ahead of time when it was in the process of deciding which packages to use and dependencies to rely on for the purposes of this project and fulfilling that prompt. So overall, I'm pretty impressed. I think things are progressing continuously in the right direction from both a production standpoint and a security standpoint for Claude Opus 4.8. But I'm curious, what are your thoughts? Let me know in the comments below. On that note, that does it for this video. If you got value out of it, be sure to like it down below and share with somebody who could put it to use. And if you made it this far, subscribe to the channel so you don't miss out on upcoming videos. Thanks for watching and happy, safe coding, everyone.
