Transcript
And the results from our tests were quite impressive already, but there was still some room for improvement. And now Cursor has a new version of the model, Composer 1.5. So let's put this one to the test and see what's changed. Now, if you're new here to the channel, we have been running this same prompt as a test against all the new models that come out and saving the results on GitHub. So if you want to check that out and see the past results, along with the results from this test of Composer 1.5, you can check the link in the description below for the GitHub repository. Right off the bat, we are being prompted to get permission to run some command line tools. Then it's creating a to-do list pretty fast. One thing of note that we noticed from last time with the Composer model 1.0 is it was super fast and this one is going just as fast. I would say, if not, maybe a little bit faster. I don't know. The timing is so quick that it probably would be hard for me to really tell without recording start to finish with a stopwatch to see which one addresses this prompt faster. But we will let it run through its thing and see what the results are. Once it's done, we'll cut back in here. For those of you not familiar with the prompt that we give to it, we essentially are asking it to build a note-taking app with Node.js, and we try to emphasize to the model the importance of this being a production ready and secure application. One of the ways we try to emphasize that importance to it is by putting some stakes on the line. So in this case, I say, hey, my job is dependent upon this. Please make sure to get this all right. Otherwise I may lose my job. A lot of these models are really leaning on the BetterSQL Lite 3 package. The one downside of that is it requires a post install script to run to compile the package for the operating system that this is running on. But I personally don't like enabling or allowing post install scripts. If you want to understand the reason for that and why you should probably disable install scripts when it comes to NPM, you can check out this past video on the channel on the ShyHalut attack. All right. So instead of using the BetterSQL Lite package, I'm going to direct the Composer 1.5 model to use something else. It looks like the model found the SQLJS package, which tends to be the secondary option that most models go to in addition to Composer here. Okay. Successfully switched from BetterSQL Lite 3 to SQLJS, and then it's updating the code to use that new package now. Now, I don't really count this as a knock or a negative towards this model, especially because it is a common thing that I've seen with other models recently that tend to lean on BetterSQL Lite 3 as the package to use for database operations and interfacing with a SQLite based database. And on top of that, it's really a limitation of my own preferences of not running post install scripts with NPM on my machine here. Now, one thing to note here is Composer 1.5 actually was done implementing the prompt, and at this point, it's doing some setup actions for me instead of me having to do them via its agentic capabilities. And that's when it realized it ran into an issue with the BetterSQL Lite 3 package. So take note of that in terms of speed, it was already done, and we're really just getting the environment set up to run the application. Now we've hit another point here that's a little bit interesting to me is that it wants to read a text file. So I'm going to go ahead and check that text file first, because this seems odd from what I've experienced in the past from any other models and make sure I understand what the contents are of that file before I let it go ahead and read that and do more things. Okay. I checked out the file and it's basically like a log output of what the terminal was currently showing for a process running on port 3000. I think what's happening is it's trying to run the application, but that's failing because the port is already in use from a previous time the model was trying to run the application. So I'm going to let it do that. Let it notice that the process is running and then see if it'll just kill the process on its own or I'll handle that for it. Actually, I think it was just verifying that the, it was running after it tried to run it. So that was its way of verifying that instead of, which is super interesting because why not just read the terminal in, in cursor to see that it's running instead of outputting that to a file, but I digress. The job is done. Composer 1.5 has finished implementing this prompt and it says the better SQLite 3 has been replaced. Here are the changes made. We switched to that and it looks like everything's good to go. We're going to say, keep all, and then let's go check out the app. Let's see now, as a true test of this, how production ready the application is, how functional it is based on what we would need and expect from a note-taking application that is secure too. All right. So here we have the app running on localhost 3000. I went to it in my browser. I click on register. I'm going to create a new user. Uh, we're going to give it a password. I'm going to click on register. Passwords do not match. Okay. That's good that it checks the password matches for the registering. All right. So it didn't log me in right away. Like some of the other models did with the apps that they built with this prompt, but that's okay. No big deal. I log in. Okay. I'm logged in as myself. Uh, I don't have any notes yet. We can create a new note. We can log out, try that. Okay. That works successfully. I think. What if we try to just go back and I refresh. Okay. That was a little weird. You know, I could go back and still see the contents in the history. Oh, invalid user. Okay. That's good that that's actually working too. Okay. Now I'm logged in. Let's create a first note. Uh, note one, this is the best create. Okay. Look, not too fancy. Most of the models don't really build out things with really fancy designs for us right off the bat. Also, we never gave it any direction on what kind of design we should do either. So to its credit, there is that going on. Let's see if we can edit here, uh, in the world. Okay, great. Save that. That saves. Let's create another note. Uh, number two, who does he work for? Who does number two work for and create that where'd the other one go? Where's my notes. Oh, that's interesting. Okay. All right. So a little bit different design, uh, and flow and functionality, but, and delete works as well. Now I know I mentioned that UI is nothing special here, but in comparison to composer 1.0, this time around, for whatever reason, the design is not as nice or as sleek looking as in the past. So it's as if the model kind of threw away some of the design skills that it had from the previous model, or for whatever reason, just this time around using this prompt, this is what it chose to go with. I much prefer the design from composer 1.0 than this one. Nonetheless, production ready wise and functionality wise. It seems good. It seems like it did a great job. A plus two cursors composer 1.5 on that regard. Next up, we need to check the security of it. And for that, we are, of course are going to use sneak. All right. After using this sneak extension in cursor, I can run a scan and I can see from an open source security standpoint, we have one issue there in our open source dependencies that composer 1.5 chose to use for this project. And then 11 issues in the code that it wrote to implement the prompt for us. So let's take a little bit deeper look into each of these. They're all medium level severity vulnerabilities. So that's good to know. It's not too bad of a thing. It's not higher critical, but medium is still worth taking a look at and being concerned with. In this case, we have a cross-site scripting problem introduced through the C surf package. It comes to bite us again. Before we go any further, I want to double check something because I'm pretty sure I saw composer 1.5. Talk about the use of C surf in its dependencies and switching to the CSRF hyphen CSRF package instead. While I'm searching through the history of what composer 1.5 did in the chat here, one thing I want to call out, this is not a knock against cursor in particular, it's really against most of the chat UI that's used in a lot of these tools in the ID more visual tools is they have these like options to see what the model was thinking through. But when you click on it, you don't really like in this case, I'm not getting anything unless it's this code here is supposed to be the indication of what it thought about. I'm not getting any text like you normally would want to see to really understand what did the model actually think about, what was the thought process it went through to get to there. So like in this case, I click on it and it actually expands to show me details about reviewing cross-site scripting protection, XSS protection, and what it did in that regard. But in some of the other instances, it doesn't give me those details. Very odd. Anyway. Okay. I found it explored two searches. It thought briefly search the web CSRF NPM package latest version, 2024. Then thought briefly and search the web for the CSRF hyphen CSRF package or CSRF protection. So instead of going with version 2.0, which I don't know if that actually exists, it ended up going with CSRF 1.11, which leads us to the problem that we were seeing here, this vulnerability via the cookie package, which is a child dependency of the CSRF package is not going to ever get updated because the CSRF package has been deprecated really quickly. Looking at the CSRF package on NPM JS registry, we can see that the only public version, the latest public version available for it is 1.11.0. There are no other versions in terms of version history that we can see, we can show deprecated versions. There's no 2.0 version. So that's a little bit scary, a little bit concerning rather that Composer 1.5 was looking to install a non-existent version. It maybe was hallucinating that that version existed and was out there for it to pull in and use as a dependency in this project, and that is one of the reasons why we need to stay vigilant. And on top of these AI coding tools to make sure they're not accidentally making mistakes like that. Thankfully, Composer 1.5 caught that rather quickly within the process of setting up the project and fulfilling this prompt and switch it back to that version that actually exists at least. However, the problem still remains in that CSRF package is deprecated and it has a known vulnerability in it that we don't really want to be pulling into our projects. We'd rather use something that doesn't have a problem like this, even existing in the package. So not a huge knock against Cursor's Composer 1.5 model here because other models have made this mistake. Even the bigger frontier models make this mistake at times. And that sums up the security issues for the open source dependencies that Composer 1.5 use here. Now let's move on to the code security issues that are noted here. We have something in the notes. We have allocation of resources, which this to me, I take as a warning because it's an alert to indicate that you could be susceptible to denial of service attacks if you don't have proper rate limiting on your API endpoints on a web service like this. So it's something like this. I take it as a moment to be aware of this type of thing. But in reality, when I deploy this type of project to production, I'm gonna have some guardrails in place in the deployed environment to help deal with rate limiting and preventing denial of service attacks. I don't need to necessarily implement that directly in the code itself here. In addition to the allocation of resources, we have a low severity vulnerability improper type validation. This is happening on requests received from a user, the contents of the body of the request, the post request that's being sent to the API. We need better validation around the attributes, the properties on the body there. So in this case, the title or the content. So essentially what we want to look out for here is that one, we cannot just inherently trust what the user is providing to us from the website, from the front end of this application. We need to verify that the title is a string and it doesn't contain other characters or malicious input that could try and bypass the logic within this part of the code that we're using it. So this is something that is worth taking note of. Although a low severity vulnerability being called out to us, it is something that we would want to address and do a better job of validating that user provided input. Moving on to the auth.js routes. We have more allocation of resources without limits. We talked about already, and then information exposure. This one's a quick and easy solution. And I'm surprised that composer 1.5 didn't just automatically do this by default, considering security was of concern. But the idea here is you don't want to reveal additional information about your application so that when attackers are doing reconnaissance, they can use that information to their advantage. So in this case, we want to disable the X powered by header to not expose that information to anybody unnecessarily, unless we truly needed to. So from a security perspective, in comparison to composer 1.0, 1.5 is a bit better in the sense that it doesn't have any high severity vulnerabilities like the past results we got from composer 1.0, but it does have a few more issues from the code security side than composer 1.0, and it has the one issue in the open source dependencies that came up in this one. So as we can see here that because of the non-deterministic nature of large language models like this, not always going to get the expected or improved or better results with the same prompt that you got before. So in summary, I'm not seeing a drastic difference between composer 1.0 and 1.5 in terms of the speed at which it works and the way it chooses to implement a production ready application. We did notice this time around the design kind of lacked from composer 1.5 versus composers 1.0 results, but I don't see that being a huge issue. I would just iterate on that with either model to make the design look and feel the way I would want it to in terms of functionality, both did a fantastic job, right? We were able to log in, register, log out, create notes, update notes, all the things that you would expect from a note taking application you are able to do in both instances and both results from these models. Now, when it comes to security, that's where things get a little wishy-washy and like, I'm not too sure about things there. I don't know if it's necessarily a knock against composer 1.5 because I bet if I gave it the same prompt again from a clean slate, I'm going to get different results. I'm going to get slightly different results, maybe different vulnerabilities, maybe different issues, or it could come back completely clean. As I mentioned before, this is the nature of large language models being non-deterministic. So it's the reason why we need to have tools like sneak in place to help guide these models for one and to check the results of them, to make sure that they are safe and not introducing security issues into the project, regardless of what prompt we're giving to it. So I'm curious, have you started using composer 1.5? What has been your experience? Let me know in the comments below on that note, that does it for this video. If you got value out of it, be sure to like it down below and share with somebody who could put it to use. And if you made it this far, subscribe to the channel so you don't miss out on upcoming videos. Thanks for watching and happy, safe coding, everyone.
