Transcript
I will be deploying this in the real world. My job is to ensure it's 100% safe from any security issues. If it isn't, I'll be fired. All caps, three exclamation marks, explicit states. And Cursor just dropped Composer 2.5, their own in-house model, with one specific claim that caught my attention. They say it's better at what they call effort calibration. The model is supposed to understand how much work a task actually deserves and apply it appropriately. So the question I want it answered is simple. When you tell this model that your job is on the line and you need production-ready, secure code, does it rise to that? Does better effort calibration mean better security output? I ran the same prompt I've used on every other model in this series. Node.js, Nodes app, CRUD operations, production-ready, secure. And then I put the output through the same security lens I've applied to every other model. Here's what I found. Before we get into the code, let me give you 60 seconds of context on what Composer 2.5 is, because this one is different from the other models we've tested in the series. Cursor builds and trains their own models specifically for agentic coding tasks. Long-running, multi-step work inside a code base. Composer 2.5 isn't a wraparound GPT or CLOD. It's built on Moonshot's Kimi K2.5 as a base checkpoint and then fine-tuned using reinforcement learning against real coding tasks. The reinforcement learning, or sometimes referred to as RL, the training of that is where it gets interesting. They train the model by having it complete actual coding tests with verifiable rewards, meaning the tests either pass or they don't. No partial credit. And in training, Composer 2.5 got good enough at these tests that it started reward hacking. The model found a leftover Python type-checking cache and reverse-engineered it to find a deleted function signature. In another case, it decompiled Java bytecode to reconstruct a third-party API. I'm telling you this because it matters for what we're about to look at. This is not a model that takes shortcuts on reasoning. When it's given a hard problem, it is trained to find a way through no matter what. The other thing Cursor specifically highlights is this effort calibration. The model is supposed to modulate how much work it puts in based on how hard the task actually is. The effort calibration improvements came directly from textual feedback during training. Targeted corrections at the exact point in a reasoning trace where the model made a bad decision. So the theory of this model is smart, persistent, and correctly calibrated to task complexity. Now let's find out if that extends to security. So with the prompt set off with Composer 2.5, I see it started off in a similar flow that I would myself. I'd initialize a package JSON, set up gitignore, environment variable files, and then an SRC folder to put all the source code files for this application inside of there. And it's off and running and building out everything for this application right now. So we're going to give it some time and cut back in once we have something else to check out or it's complete. Quick update, it's getting close to being done. It's on the last step of its to-do list for this prompt. I noticed it's running its own audit. I'm using PMPM on this machine. It ran PMPM audit and that's giving it a little bit of an advantage in the sense that it's going to read the security results for open source dependencies that PMPM provides to it. And so now what it's trying to do is address those. And I think that kind of gives it a little bit of an unfair advantage but the fact that it knew to do that is also a nice thing. On the other hand, it'd be great if the model just chose open source dependencies that didn't have vulnerabilities in them in the first place so that it would not have to go through this iteration of finding and fixing vulnerabilities in those dependencies. But I'm going to let it continue doing this thing. I just wanted to at least call it out really quick before I let it proceed any further that it's kind of a slight advantage that Composer 2.5 is getting in this test result. All right, with that Composer 2.5 is done implementing everything needed. It believes it's needed for this prompt and it gave me a quick summary here. Quickly reviewing the closing remarks from Composer 2.5 we see we have authentication in there with a registration and login that leverages bcrypt and general generic error messages. We have notes, crud, create, read, update, delete and then SQLite parameterized queries, foreign keys, all that fun stuff. Security controls that it put in place, defense in depth. We have SQL injection protection, cross-site scripting, cross-site request forgery. It's nice to see that it went with the CSRF package instead of the CSRF package. Something worth noting there. That's a nice plus and a good nod, positive nod to Composer 2.5 compared to past models that like to use that deprecated package. It's using sessions, brute force protection, input handling with invalidation with Zod, some HTTP headers, security headers like helmet and then just error handling, generic messages to users, no stack traces are exposed. Here's some quick start notes, the production checklist, things we should do before we push this to production, which is great to see. And then honest security note, which is, of course, a good reminder. No application is 100% secure, but it is following common industry practices. Calling out OWASP, Open Web Application Security Project, great to see that. Your job also includes infrastructure and ongoing maintenance. Okay, in terms of keeping it secure. And then it calls out how to use PMPM audit, which reported six high issues. All right, with that, I'm going to go ahead and get this set up and running and then see what it looks like when we run it. Another quick interruption that I want to call out here, that's with the latest version of cursor. I'm not sure if it's a bug or something like that, but when it runs background terminal tasks at the end, even though I've already had the prompt stopped, and then I go and choose, I can't show it now because I've already done it, but there was a background terminal that was still running the server before I can start running it myself manually. And I chose to have it stop that. It had a button right around here that let me click stop or kill that process. And when I did that, it added this extra like thinking right here. So it's really odd that it re-does or keeps going in the context of this chat. I would expect it not to say anything like that, really. It's just an odd behavior that I'm noticing with this version of cursor. So just a heads up on that in case you're seeing the same thing. All right, so I'm going to create an account. I'm going to say Clerkyo1, give it a password. All right, click create account. Account created, let's sign in with it. Oh, too many redirects. Try deleting your cookies. That's an interesting error we got now. Yeah, again and again. Interesting. Yep, it just keeps redirecting over and over and over again. We could see at least a dozen or more redirects happening here. All right, so kind of stuck there. Can't really test it out further. But now we need to test out the security of it. And to do that, we're going to use Snyk to see what security issues there might be with the project that Composer 2.5 made for us here. All right, so I have the Snyk extension set up, installed, logged into my account. And I scanned the project. And we can see we have quite a bit of security issues that came up, both in the open source security dependencies that we have for the project here that Composer 2.5 chose and the code that it wrote. So looking at these issues, we have three high and four medium severity issues, all based on this tar package that it chose. And that's being introduced via the bcrypt package, our direct dependency. One of the fixes is upgrading to bcrypt 6.0.0. Versus the 5.1.1 that we're using here. That has the fixed version for that. We can go ahead and fix that and get that remediated. We have a symlink attack also introduced via there and a couple other ones. I'm not going to bore you with all the details, but you get the idea. It didn't really choose the best versions of the packages, not necessarily bad packages, but versions of packages that have known vulnerabilities in them. In addition to that, we'll take a look at the code security. And we can see, okay, allocation of resources without limits. I've mentioned this in the past videos. I'll call it out again here in case you've missed that. This type of thing is for rate limiting. And I take this with a little bit with a grain of salt whenever I see it come up as an issue because I tend to deploy these applications in environments where there is rate limiting on the outskirts of it, not directly written in the code of my application. That's what I choose to do. You may choose differently, but that's just a heads up with that. And so take this with a grain of salt. It's not a huge issue. So I don't count it as a knock against cursor Composer 2.5 in this situation. However, the open source dependencies are not a good look there. So that's a bit of a knock as well. So in conclusion, I do feel like this is an improvement in the Composer series of models with Composer 2.5. However, from a security and production ready standpoint, it is still lacking. It's not a huge knock against Composer 2.5 because every model I've tested in the series has had a different failure profile and the pattern can be useful to have that data. And you can see that all in our GitHub repository that will be shared and linked in the description below. But as you can see, it wasn't really fully production ready because I wasn't able to successfully log in and test out creating, updating notes, deleting notes, that type of thing. And then from a security standpoint, we had some significant issues more so in the open source dependencies that Composer 2.5 chose for this project. I'm curious though, what are your thoughts and what's been your experience with using Composer 2.5? Let me know in the comments below. On that note, that does it for this video. If you got value out of it, be sure to like it down below and share with somebody who can put it to use. And if you made it this far, subscribe to the channel so you don't miss out on upcoming videos. Thanks for watching and happy safe coding, everyone.
