Transcript
with GPT 5.5. Two brand new flagship models, both promising to be the smartest coding AI on the planet. So which one do you actually trust to build something real? We did what any reasonable developer would do, we gave them the exact same prompt. Build me a production-ready, secure note-taking application. No hand-holding, no clarifications, one shot. Then we put both apps through the full gauntlet. Does it actually run? Does it look the part? And most importantly, does it hold up when you run it through security testing? So stay with me because the results may not be what you expect. On the left-hand side here, we have OpenAI's GPT 5.5 at extra high reasoning selected there. On the right-hand side, we have Opus 4.7 on the max effort level as well and thinking. We're going to send the same prompt into each one at the same time and see what they produce and then test the results of those. So here we go. Sending both in. All right, so both models finished at this point. Opus 4.7 finished a little bit earlier than GPT 5.5. I want to say it was right around 12 to 14 minutes it took for Opus 4.7 to complete. And then about two minutes after that is when GPT 5.5 completed. Throughout that whole process, I was getting more updates and visual indications of what was happening in the progress that was being made on the Opus 4.7 side. I don't know if that's necessarily the model doing that or Cloud Code extension in Visual Studio Code here that's providing that to me. Whereas on OpenAI's GPT 5.5 side, I think their extension, their harnessing around the model limits what's visually shown in the chat screen here with that model. And so it was kind of abstracted away from me. And then all of a sudden I would see bursts of changes that happened from that model working on this prompt. So it's something if personal preference, if you care to see that type of thing, or if you care of things being abstracted away, you might value one over the other just for the sake of that. As a result, both have taken the security seriously with the prompt of this application. They gave us summaries at the end of what they did. Opus 4.7 had a more technically verbose to-do list of what it was going to do to fulfill this prompt. Whereas GPT 5.5 had like four to-do items that really were rarely updated and shown making progress because of what I was talking about before. Things seem to be abstracted away from us as the users of that within the extension here inside VS Code. On that note, let's check out Opus 4.7 first and see how it did with this prompt. All right, so Opus 4.7 built out this snazzy looking, nicely styled dark mode application for us. We have login capabilities or we can sign up. I'm going to sign up with Clarkio. All right, request rejected. 403 error. Sign up. Let's try again. Keeps getting rejected. Is it an issue with cross-site request forgery or something? Invalid CSRF token. All right, so something in the way Opus 4.7 implemented its cross-site request forgery mitigation is not quite working in here for us to test this any further at this point without maybe prompting the model to make changes with this issue or us going in and tinkering with the code itself. So I'm going to leave it as is because I don't want to do any hand-holding or anything like that to customize this further for the purposes of a fair analysis of the two. But outside of that, it is nice looking. I feel like it probably would have the CRUD operations done properly. It's just a matter of getting a login set up and working here. All right, next up is to test the security of the application that Opus 4.7 put together, both the dependencies and the code that it wrote. So we're going to use Snyk for that using the VS Code extension that Snyk provides. And we can see when it comes to the open source dependencies, we have no security issues. So all the dependencies that Opus 4.7 chose and then used npm audit. So that's a little bit of an advantage that it took there, but it was smart enough to know to do that. It went and fixed and updated the versions of those dependencies so that they didn't have any security issues in them. And therefore, Snyk is not finding any issues with those open source dependencies. Now, when it comes to code security, we have a few things here. We have nine open issues. We have a high severity one and a couple of medium severity. So let's check into these a little bit further. We have hard-coded non-cryptographic secret. So in this situation here, we have a dummy hash. Avoids leaking which usernames exist via response time differences. Pre-computed hash used to keep timing roughly constant for unknown users. This is indicating to us that we should not have this hard-coded like this. We should use things like environment variables or secrets management solutions that can inject the value for this that we can use so that it's not committed into our code and pushed into a branch that somebody else might have access to and visibility to and expose this secret there. So regardless of this, we really shouldn't be hard-coding anything that could be potentially sensitive, especially when we're doing cryptographic operations like this. So we should use something like a password manager or environment variables at the least so that it's not hard-coded in the source code of the project here. So that's one thing there to look out for. The rest of these in the auth.js file are allocation of resources without limits. The issue here that we're being alerted to is the potential for denial of service attacks essentially. If we don't have proper rate limiting around these endpoints, we could potentially be susceptible to a denial of service attack. I take these with a grain of salt personally because I tend to deploy my applications in environments that have resources available built into them to help with rate limiting. So I don't necessarily have to write that into the code myself, but something to be mindful of. Moving on to notes.js, we have more allocation of resources without limits there. Ooh, we have something different here. Use of externally controlled format string. Let's see this. This is a medium severity. This unsanitized user input from the request URL flows into error, where it is used as a format string. This may allow users to inject unexpected content into an application log. Very interesting here. So where is this happening? On line 21 here, where Opus 4.7 added this console.error function, and it's passing in the request method, which is not necessarily a bad thing, but the request original URL could have some nasty stuff in there that might be an issue. That's definitely something to look out for. And a little bit of a knock towards Opus 4.7 here. All right. And last but not least, we have a cross at request forgery alert from Snyk here around this application setup with the Express. So typically what Snyk is looking for is the use of the Csurf npm package, but that has now been deprecated. And the Opus 4.7 model has implemented its own cross at request forgery middleware, which we saw isn't exactly operating successfully to the fullest extent. It is protecting the API endpoints that are doing state changing operations, like when we're creating a user. However, it seems like it might not have implemented the cross at request forgery middleware properly enough so that when we are doing things the right way in the application through the UI, it's not allowing us to create a user as we saw. So something to look out for there. And another slight knock against Opus 4.7 in its implementation of this prompt. Now let's move on to OpenAI's GPT 5.5 implementation. All right. So I have OpenAI's GPT 5.5 implementation of this prompt up and running on localhost 3000. And we can create an account here. Let's try this now and see if we get further than Opus. Yes, we do. Okay. Already a plus one on GPT 5.5 side of things. So we have private notes. Your notes are here. You can create a new note. Say note one. This is my note. We create that note. Let's create a second note. Note number two. Testing one, two, three. Create that note. And if we want to edit note one. Edit it now. That's working as expected. Okay. If we want to delete note two. All right. And if I sign out and I try signing in with the wrong password. Invalid username or password. All right. Everything's working nicely. So functionality wise, OpenAI's GPT 5.5 seems to be in a better position right now than Opus 4.7 because we're able to sign up with a new account and do all the things you would expect as a result of the prompt we gave to it. So now let's test out the security of the implementation from GPT 5.5. All right. Again, we're using Snyk's extension inside of Visual Studio Code. And we can see that the open source security, the dependencies that GPT 5.5 chose to use came back clean. There are no known vulnerabilities in the direct dependencies or any transitive dependencies of the packages that chose to use for this project. However, when it comes to code security, we can see we have some issues, albeit a total number less than what was being reported on the Opus 4.7 side. So here we can see we have high severity issues and low severity issues. In app.js, we have cross-site scripting vulnerability, which is different from what we saw in Opus 4.7. Here we have the potential for unsanitized input from the HTTP request body flowing into the send where it is used to render an HTML page returned to the user. It could result in a cross-site scripting attack here. So yeah, look at that. The HTML is straight up sending that back. All right. So with this one, it's not a good look. I don't want cross-site scripting in my application. And this is something I could test out later on, sending in a cross-site scripting payload to see if it does indeed execute in this context that's being leveraged here or available to us in the application here. In addition to that, we have the typical cross-site request forgery. We talked about that in the Opus 4.7 results, similar situation here. I'm not going to count that as a knock against GPT 5.5. And then we have some low severity ones. And this, something to note here, is it's happening in the test files, use of hard-coded passwords, which we can then choose to create and ignore if we wanted to, because we need to let Snyk know that, hey, this is a test file that's doing this. It's expected here. It's not part of the production running code. We can let ignore this safely. So really, it's just the two issues and one of them being the major issue, the cross-site scripting vulnerability here. So with that, who do you feel is the clear winner here? Is it Opus 4.7 or GPT 5.5? Let us know along with your reasoning in the comments below. On that note, that does it for this video. If you got value out of it, be sure to like it down below and share with somebody who could put it to use. And if you made it this far, subscribe to the channel so you don't miss out on upcoming videos. Thanks for watching and happy, safe coding, everyone. Transcribed by https://otter.ai
