Transcript
I think it's Claude, but what do you think? I'm very biased and I guess after your like, your segment here where you talk about your experience with AI, I wanna show everybody some benchmarks and one particular benchmark that was shown to me last week that I really enjoy, that maybe give some like more measurable data to why things are better than others. And I think for the audience here, that might be something tangible that's like, okay, let me actually see why it's better versus a chat GPT was bad. Yeah. Co-pilot was slightly better cause really under the hood, it's just opening AI. Yeah. And then Claude's different. Yeah. Okay, so my assignment this week was to dive into Claude. And so I talked to Josh about it and we talked about this in the pre-show a little bit. Josh really doesn't get nervous about much. It's just the person that he is. So part of Claude is you download the desktop application. And when you first sign up to a PDQ as an enterprise license and he said, Tara, do this, it's gonna say, do you wanna download to do co-work? So you can do the chat, which is very similar to chat GPT and you can do co-work and then you can do code. Yeah. As many of you know, I've been learning PowerShell and I've been learning, taking my daily lessons with the chat GPT. So I thought, well, I can just continue to do that in chat. But if I really wanna dive into the power behind Claude, I need to do co-work. And I immediately went, I can't do it. So then I chatted with Josh and he's like, just do it. Don't be such a baby, it's fine. If it deletes your OneDrive, we'll restore it. And I was like, what? He really said this. Yeah. And you actually said. When you set up co-work as part of the process, you give it two settings, I'm going off memory here. One of them is code execution and the other one is file creation. And in file creation, that also means modification deletion. Correct. So you're giving that access to any connected service, including your user profile on your device. Correct. So everything that he just said absolutely terrified me, but he was like, go ahead and do it. Yeah, some people have lost data, but do it. Now, I am not the type of person that can hear that and be like, that's fine. I can do it. No. So I didn't go forward with what Josh said. I was very cautious. So I watched a total of five videos. All the videos were 45 minutes long. I wish I was lying, but I'm not. And I was very cautious in how I progressed. So I learned some things and Drew's gonna attach what I felt like were the most helpful solutions and guides that I wanted to give you. I liked these ones the best. And the very first thing that I did was to create, I think they call it the AI solutions or AI master list. Basically what it is, is you tell Claude how you want it to behave. You give it an identity, you give it settings. You tell it who you are. You say, these are my goals and objectives. This is what I want to accomplish. These are the things I want you to automate. And most importantly, what I felt was most importantly, these are my non-negotiables, aka you can never do this. So I probably spent two hours on this. I have a question for you. Okay, why am I so crazy? I don't know. No one knows. No, no, no, no, no. So did you create this system file yourself? No. Or did you use, or did you take this as a template? This is a template that I got from this company. And then give it to Claude in chat and say, help me work through this. No. To give back to you. I did not. I took it from this company that I watched. I actually watched videos from a couple different sources. I took this exact template and then I made it work for what I'm going to do. So I heavily edited it because I'm me. I didn't like some of the things. I first put it into PowerShell. I edited a bunch and then I was like, I don't like that. Yes, I'm crazy, you know me. You're not crazy. I'm just, for my co-work file, I took the same approach except for I gave it to Claude chat and said, hey, Claude, I want to work through this file with you. I need to make this. Ask me questions that will help me get to a reasonable point to restrict yourself. That's essentially what it is. So I'm using the tool to create the thing to like bootstrap itself. Yeah, that's smart. I'm innately distressful. You are innately trustful, I think. And actually maybe not trustful, but I think you're more devil may care. I'm a nervous person. So I wanted to put the restrictions in right off the bat, but there's two different ways to place the restrictions. There's the global settings and then you can build automations or skills is what Claude calls them. So you can actually put those, they're called .md files in the skills. Okay, so you can do it in two different places. So that's where I started. Rockstar is like, Claude, please create restrictions for yourself. Yeah, I mean, that's essentially what I did. Yeah. Okay, so after I put in the global settings, then I started, I thought, let's try it out. And I did my first lesson in PowerShell. So I did have to give it access to my files and you can say, you can give it as many access, as much access or as little as you like, right? It works best if you create a folder on your desktop or anywhere actually, and give it access to the file with which you will be working. So for PowerShell, I have kept a document of my lessons. I took that document, put it on the desktop and said, we're going to be working here. I have been working 35 days. I've taken 35 lessons. Please read it. And it's like, took 10 seconds. Great, here's your lesson for day 35. I've taken two lessons so far, vastly superior. Oh. To ChatBeech GPT. Wild. Vastly superior. Wild, and I'm gonna show you why when we dive into the data. Okay. Why specifically Sonnet 4.6 is beating things from OpenAI right now. Okay. For specifically that reason. Yeah. So after I saw how much better it was in the lessons, the first automation I set up with Claude is I said, I want you to do a lesson for me every day during this time, at this length, after the end of the lesson, based on the questions that I ask you, create a summary, add it to this document in this location. And then you tell it the location that you're working in. And then it's like, hey, here's a summary. Can you proof it? Okay, I'll add it. Oh my gosh. Love it. It's great. It's great. Here's like, here's like maybe my meta thoughts on AI generally. It's gonna do two things. People that are already efficient are going to become more efficient. People that are intelligent and able to use the technology are going to become more like smarter, more intelligent. Yeah. Do two things. It also is going to cut both ways. I think people that generally want to just believe the things they've seen or like have confirmation bias, it's going to empower them to be more maybe flawed in their reasoning. So it's gonna do both things. So it's gonna create a divide. And I really wanted to bring that up because as you know, I'm a very careful person. And that's why I was very careful in setting it up. And I put all the guardrails up in every automation. I'm gonna call it skills first, because you can do a skill and then you can automate that skill. I have put in the markdown file for every automation. I said, do this, don't do this. So my non-negotiable. So this morning I actually made an automation to scrape my email every hour, create what I'm, I want them to create a draft, him to create a draft for any email I haven't responded to, let me approve it before it sends. But that's the key. Don't send it unless I tell you to. And the same thing for Slack. So in Anthropic and Claude, there are skills. And skills are like repeatable, automatable processes. You have very specific guidelines for it. There's very little generative work involved there. And then you've got projects. Projects are more like system declarative prompts, but still generative work. So choose the right tool for the right job to get the best results. But I think that you have to tell him what you want to accomplish. So if you put in garbage, in my opinion, you'll get garbage out. 100%. Yeah, if you're very, very clear with him what you want to accomplish, much better. 100%. And he'll argue with you. So part of my global settings were, because one thing that drives me crazy, and it might drive you guys crazy too, I hate the emojis. I hate the really big bolded text and all the outlines. So I'm like, never do this. Never give me emojis. Never give me this bold. Never tell me I'm right when I'm wrong. This is the language I want you to, you can tell it tone. So I really enjoy that about Claude thus far. So, but anyway, do you want to show us why? Let me show you why Claude pushes back on you. Let me show you why Claude is better at reasoning, right? Okay, so I was shown this benchmark last week. I had not learned of it. This is amazing. So this benchmark ranks the individual models based on their likelihood to push back on the user based off of like trying to compare two separate things. And when you try and compare two separate things or the AI model is agreeable, that's how you get the hallucinations. That's how you get the garbage, right? So if we look at the top of the list right here, so in 91% of prompts, there was clear pushback on the user of, hey, what you're doing doesn't make sense. 6% of the time, Sonnet 4.6 would push back on a partial and then only 3% of the time does it accept nonsense. So if we look at that and we look at your experience, this is why Claude, because Sonnet 4.6 is the default model, is pushing back so hard on you. Of like, that doesn't make sense. Let me challenge, let me ask questions rather than just the blind acceptance. As we scroll down this thing, it gets to be wild, right? So even GPT 5.4, right here, let's just scope this directly to OpenAI. 5.4 is less than half the time. If you give it two wildly separate things, it'll just be like, yep, you're right. You're so awesome. It is, just a quick question and maybe this doesn't answer it, but does it change if you change it to quick or deep thinking mode? It does, it does. So let's scroll down. So we're gonna scroll down here a little bit further to here. Here's this detection over time. So if you wanted to compare the three major providers here, remember Copilot sits in that green line as well. Does thinking harder help? Is the question that I think you asked. So this is a green rate. So green rate is essentially this thing that says I'm effectively using computing to get the best result where like a red rate is I don't care how many resources I burn through to get to the result, okay? So if we look at just Anthropic up here, you'll see that it's green rate even on 4.6 high and 4.5 high are slightly higher, slightly higher. And it really starts to matter with the task that you're giving it, right? The other thing to keep in mind here is what is the market that you're getting into? Is it physics? Is it creative? Is it mathematical? We're starting to see a differentiation in the mainstream AI that people are picking up on. Bigger number, more expensive model does not necessarily mean better, right? Historically, that's been the, oh, it's bigger, it's better, it's faster, it'll be smarter. You'll actually get worse results if you pick the wrong model. Interesting. Right? So something like an Opus, which up here is still by Anthropic, right? We come back to Anthropic. Opus is pushing back, but only as an overall. Now, if we start to scope these out into software, finance, legal, medical, physics, here's that mathematical part of this, where in the other places where you can have some opinion, maybe legal, maybe finance, they start to fall back and they change. Software, more mathematics. Opus probably isn't the best choice for you. And you can come and you can pick your provider in here. So you can pick if it's GPT, if you're a Google shop, you can see which one it is. And you can start to break all of these down and pick the right model based on the task that you're doing. Because thinking longer, thinking harder, more tokens doesn't necessarily mean better. So everyone should take a look at this. This is a benchmark, and this is a benchmark on pushing back for comparing two different things. That is just a way of looking at this. There's lots of parameters, lots of variables. But if you don't want your AI chat assistant to be this agreeable, you're so smart. Pick something like this, because it'll poke holes in your logic. Yeah, love it. Well, okay, so here's my final assessment. When we started this, you told me I could keep both. Yep. ChatGPT and Claude. And I was like, you know what? I'm gonna keep them both. Because I kind of like, after a week, oh no, I'm breaking up with ChatGPT. I don't want it. I love Claude. You know what? It's like the race, right? Between the major providers. Who's gonna do what? It becomes like this context thing. So wrapping this back up into the sysadmin frame, I actually posed a question to some of the leaders at PDQ today. Of like, hey, in the last couple of weeks, IT's seen this big migration to Claude, in ditching OpenAI and ChatGPT. Is there any concerns with consolidating on one? And the two themes that immediately emerged from that is, yes, except for I have the shared custom GPT, and there's a handful of them throughout the organization, that I'd have to migrate over. Because we've done that, I now have like general purpose users that use both, that I have to license for both. The other thing that came out of that is, okay, well, Claude and Sonnet and Opus seem to be winning right now. But what happens in six months if something comes through here, are we gonna just do this migration back and forth? And I think that's the thing to consider for like administrators, is not only your use case, but your organization's use case, and how are we gonna think about tackling that problem? Yeah, because actually, and I reached out to you with this, there are a bunch of guides on how to transfer your data. And there was one that you could transfer all of it. And it seems like ChatGPT has disabled that functionality. Yeah, I mean, they have to defend themselves in their space. Essentially, it's a giant, all it is is a prompt from one to the other that says, tell me everything you know about me. And then you paste it into the other one so it can learn. Yeah, I gave up. I can tell you there is one area where ChatGPT still reigns supreme, at least in my testing. It's the audio interface. Really? If you're having just a conversation back and forth, ChatGPT is more responsive and much more natural at conversation. Thanks for watching this segment from PDQ Live. If you like this, you'll love the full show. Check it out every Thursday at 10 a.m. Mountain. Oh, and like and subscribe, please.
