Transcript
How are you? Thank you. Hey, Angelo. Thank you. Great to see you. Great to see you. You have something to show us? I think we have a few things to show you. We've been cooking. So, Manav's right. For a single, small, local repo, agents shine. But that's not the stack our enterprise customers are working on. With large monorepos or multi-repos, agents break down because they're trying to chain together customers. So, for example, if you have a single, small, local repo, and you're trying to chain together customers, you're not going to be able to do that. You're not going to be able to do that. You're not going to be able to do that. You're not going to be able to do that. But that's not the stack our enterprise customers are working on. With large monorepos or multi-repos, agents break down because they're trying to chain together contacts by calling dozens of tools and running thousands of API requests. But the data quality suffers. So we had to reimagine context at scale. So, Angelo, how about you tell us how we solve this by building Orbit? Thank you, Meg. And just a touch on that. GitLab itself is a classic example of a mega-monolith repo and thousands of repositories within our own organization. So that's where we had a pretty crazy question. What if we took all of GitLab's data and turned it into a graph that agents could query directly? And we did that by building a highly distributed system in just three months that essentially is an indexing engine. And as I show you here in the schema, what we can do is we take... Actually just the other week, we've been able to index 160,000 repositories into code graphs. And as you can see on the right here, we take those graphs and we marry that data to the rest of the software development lifecycle. And that unlocks a variety of different questions, which we'll jump into. But one thing to touch on here is we've built this for scale. So all of this can be indexed, namely 500 million nodes and 2 billion edges in just 15 minutes. So to touch on that a little bit more and how this improves agentic outcomes, I'd first like to ask everybody, has anyone ever gone through the painful experience of refactoring a code base? I don't know. Maybe a show of hands? Me too. It's not fun. So with that, let's pull up a little bit of an example of what Orbit can do for you. So I'm going to kick off this run here for what we call the Orbit benchmark. And as you can see, we'll go into a little bit about what Orbit does. We've got this Orbit benchmark here. And we've asked a very relatable question to GitLab itself. So at GitLab, we're actually currently exploring decoupling authentication authorization within GitLab itself. And as anyone knows, if you have a mega monolith, decoupling such a critical service is a huge pain. And if you're doing that manually, that'll take months. And if you're doing that with agents, there's high risk involved, right? And in this repo, how many files are there? There's around 50,000 files, and it's millions of lines of Ruby code. And so if we're going to make any change, we want to know what's going to be affected and how. So if we go back to the benchmark here that's already running, we've asked a prompt saying, can you get the complete authorization class hierarchy and all of its front-end consumers within the GitLab monolith? And remember, this is 50,000 files. So let's jump into a little bit what's going on here. So quad code without Orbit is doing the normal thing that you would expect any agent to do. It's searching through all 50,000 of those files. It's using the classic tools like grep and text search, and essentially assembling the entire picture from scratch. On the right, the agent has access to Orbit. And what we've done is built a universal code-indexing engine that indexes over 11 languages into a unified graph that essentially acts as a pre-built map for your agents to query directly. And what that means is that agents are able to write their own queries here, as you can see, and get back the entire authorization tree in just a few hundred milliseconds. And so that effectively allows you to pretty much answer most of the questions that you would normally ask an agent, but get that answer back a lot quicker with a lot higher quality. And as you can see, the agent on the left is still running. And here's the output report with all the authorization classes. And to increase accuracy, we've leveraged a lot of different techniques like SSA and various compiler techniques. And Angelo, I think what I saw when you scrolled up, the agent on the right with Orbit is already done cooking at just a minute and 15 seconds. We're still chugging along on the left. But I have the most important question of all, which is, tell us about the data quality. So that is the most interesting part about this whole experiment that we've been running. So as you said, it's still running, and so we won't bore everybody with the results there. But with this previous run, you can see that in just one minute, we completed the result. And we just saw that with the previous run. And it took over 11 minutes. So this is, Manav, you and I were just talking about this. This is you and I, if we're coding, we can stay in the loop on the right here. And on the left, as a developer, maybe I'll go get some coffee. I'm going to break your flow. Yes, we don't want to lose our flow. So with that, the last thing I wanted to touch on that's very interesting about this benchmark is the accuracy. So we did something to measure, in a deterministic way, the output of this report. What we did is we took the actual Ruby on Rails runtime. We generated a script to get that same class hierarchy. And as you can imagine, both of these cloud codes don't have access to that runtime. So what we're doing is comparing the classes that come from the actual runtime itself with the results from the agent. And we've instructed the agent to output those classes. And what's very interesting is it's not as complete in the amount of rules that it found from the authorization classes. But Cloud Code without Orbit actually hallucinated 1,000 more rules as compared to Cloud Code with Orbit. That's 1,000 more rules that you have to manually go make sure that it got right. Yep, and so that's what we're talking about with agent trust. So with that, we can imagine how this is super useful for a developer like myself on your local machine. But we didn't want to just stop there. We wanted to empower all of organizations and enterprises with this technology. And with that, we built a service. So Meg, why don't you show us what else we've been cooking? Yeah, so as Angelo's alluding to, we didn't just stop at indexing the code base. We indexed your entire GitLab instance, which means you have access to all of your rich GitLab SDLC data. So one of the perfect examples is a pipeline analysis. For all the DevOps and platform teams in the room, you might have heard that one in three CI pipelines fail, which can really add up at scale. And in the paradigm before Orbit, you could really only analyze your kind of pipeline health at a single project at a time. But now with Orbit, because we have these traversal and aggregation abilities, you can understand thousands of pipelines and thousands of projects and their pipelines at once. So another one of our goals with Orbit is to make Duo Agent Platform even more capable. So we're jumping in here to agentic chat, and we're going to ask a pretty heavy-hitting question here. We're going to ask the agents to deep research our most recurrent failing pipelines and their jobs over the last 60 days. So in GitLab Org, that's 8,000 projects, and it's 12 million pipelines. This is a huge question that we're asking. So we have two instances. We have GitLab Duo Agent in the old paradigm on the left with access to just the GitLab API, and we have the same GitLab Duo Agent on the right with access to Orbit. And I'm going to fast-forward into a completed output and show you what we're looking at here as well. So on the left, let's zoom in for just a moment. The agent says to us, I'll be straight with you, I can't do it. And this is a limitation not of GitLab Duo, but of all agents today, because they don't have access to Orbit. To achieve an analysis like this, it would have to make tens of thousands of API calls, and the API would just time out. But we have Orbit now. So our agent on the right is actually traversing the whole graph and aggregating the job failures. It's understanding the projects, the failed pipelines, and the common field jobs underneath them. It's giving us a CI compute cost attribution and looking at those common field pipelines, and then ultimately taking us to the most important question of all, which is, what do I change to resolve this? So as we continue to see the agent with Orbit, it gives us the shared CI template hotspots and points us exactly to what I'm going to work on today, which is resolve these. And with this resolution, we're probably going to save developers a few headaches and maybe a few dollars on CI compute. Amazing. This is just one of the possibilities that Orbit makes possible. APPLAUSE We've been doing a lot of cool stuff. And Angelo, I know our engineers have been asking some crazy questions because Orbit can go further than any agent ever could. How about you tell us some of those wild things our engineers have been looking at? Thanks, Megan. That's a great question. And we as a team sat down, and we've been just playing around with it and asking some of the most wild engineering questions that you would ask when you can ask anything about your GitLab instance. And just some examples here that we pulled up off the cuff. One of them is, find all the dead repos across GitLab Org, or find all the critical services within the fulfillment department and who maintains them and who is the expert in those services. And then one of the craziest ones was we actually took the call graph technology that we built and indexed all five years' worth of repositories, then used the graph that's available for the SDLC and did a cross-comparison. And we were able to basically get a trend line of various security fixes throughout GitLab over the past five years. So basically, whatever you can do with your imagination is the limit. And like Manav said, if the nervous system is what lets your body act coherently, then Orbit is that for your entire software organization. And until today, your agents have been definitely flying blind without one. And agents in the GitLab Orbit just work a lot better as a result. Yep, we're sending them to Orbit. Amazing, amazing. Hey, thank you. Great job, Antolo.
