Transcript
generative AI applied to your infrastructure. So my name is Roxanne Fischer and I'm the CEO and co-founder of AnyShift.io. I'm a former AI researcher who worked for like doing research for AI at FinTech company and Samsung and now building the AI context for your infrastructure at AnyShift. Today we're going to speak about gen AI applied to DevOps in four parts. The first one is going to explain how actually generative AI works, so how those large language models work. To then speak about their current limitation applied to your infrastructure and DevOps. To then actually make the distinction between different type of generative AI because it's a huge work world, sorry, that encompass a lot of information but there is a difference between generative AI and synthesis and we are going to see how. The last part it's about the future of generative AI applied to DevOps and how a context is so important to make things work. So first of all generative AI, AI models, how does it work? So what's an AI model? An AI model is a set of interconnected nodes which are already created to mimic human brains. Those nodes are connected through different layers, through weighted connections. So what you can see on the right would be a small simple architecture of a neural network with different layers from simple ones, so the green ones, to more complex one, the blue ones, to even more complex one, yellow. The way it's going to work is that each layer is going to understand some patterns that it's going to encode and the more you go deeply into a neural network the more complex those patterns are going to be. The way it works in terms of understanding those patterns is that you're going to train those AI models. Initially it's going to be very dumb and those layers of interconnected nodes are going to be all set to zero. Basically you're going to initialize something you don't know what it should recognize so you put everything to zero. And the way your model is going to be trained is that you're going to say okay I know it's dumb but I know it can learn and it can learn on like huge amount of data. Those amount of data can be text, images, code bases, the entire internet, books. And so what you're going to do is that you're going to feed those algorithms, because at the end an AI model is an algorithm, with those data and you're going to train those models on those data and update those weights to actually understand the patterns of those data. And so you're going to create an iterative loop where you're going to do it once, twice, three times, ten times to update progressively those weights to understand better and better what the pattern behind the data is trained on. The core principle about this training is that you want to encode some information in the right manner. All those AI models are going to actually understand and encode information in the right manner and then be able to create some tasks. You can for instance have AI models that understand that an image is an image of a cat, so those are labeling models, but you have also models that are going to generate some content. So for instance JGPT, large language models, which are going to actually create some text based on previous information. So let's focus on a concrete example to see how actually those models are capable to encode the information and do a transformation based on this information. So here let's say you have some text. I like red apples. What you're going to do is that you have an AI model which has been trained. So it has been trained on a large amount of data and it's capable of recognizing patterns. So what it's going to do is that it's going to encode the pattern of the I like red apples into a mathematical representation. So with these weights that we've been talking about, which have been trained, it's going to train, it's going to encode this text into a vector of information. A very dense vector of information which is actually embedded in a latent space. In this latent space, you want to make some transformation. This is where you can actually make some transformation from input A to input B. Let's say here you want to add the notion of queen. So you're going to transform this mathematical representation from one vector to a slightly modified one because you added a new information. You can see here that the mathematical vector has been slightly changed. Once this mathematical representation of I like red apples have been modified, you're going to decode this new vector and here you go, you get a new output based on the previous one. So how does it apply to code generation? Your large-angle models that are going to generate some code are based on the same principle. They're going to encode and decode some text to create new content. So let's say here that you're actually creating some platform resource and you want to use a copilot, a github copilot to help you go faster in your day to day. What copilot is going to do is that it's going to take into input what you began to write and it's going to encode this text into this vector. Based on that, this model is going to continue and the transformation it's going to make is that it's going to say okay, I got this vector. The most likely next token, next information I should get based on this token should be between those three, those four, those five outputs. So let's say that here it will say that 90 percent of probability that this vector is actually predicting AMI 1, 2, 3, 4, 5 is 90 percent. You have eight percent of AMI, ABCD, EF 1, 2 and it continues like that. And so your model which is a probabilistic one, it's going to create some probability, is going to actually take the next token based on the context and take the one with 90 percent probability which is AMI 1, 2, 3, 4, 5, 6, 7, 8. After that, it's going to continue. It's like a sequence based model and so this specific AI model is going to encode decode information but so it's going to do it progressively and so this new text that has been generated is itself going to be encoded in a new vector. And you continue like that. This new vector is going to be taken and the model is going to predict what is the most likely next token to be generated. And so in this case, you have 55 percent chances that it should be like a T2 micro because your model which was trained on large amount of data, so a lot of T2 micro in this context specifically. And so on and so on. And so your model which is applied to their form code in this case is going to progressively predict next tokens based on all the mathematical patterns it has learned based on its previous knowledge. So now we saw that those LLMs are trained to understand patterns and based on that, predict new tokens, new words, new code generation. But there's also a limitation to that. And very often, like new models tend to be amazing. You have chatGPT, 4.0, 0.1, 0.3, Gemini, which give incredible results and improve efficiency by 10x factor. But also sometimes it's kind of frustrating because you don't have exactly the kind of results that you would like to have. To understand why sometimes the result is not as good as expected, the first reason is that you have the data, those models that are trained on, which are not as good as other datasets. So the first thing to take into account is that AI needs a lot of data to be effective. So I said earlier, those models to actually learn patterns to be recreated and to actually be able to make some transformation on them need to be trained on extensive datasets. The entire internet basically for the latest models. Those datasets need to be super diverse for those models to be able to actually create some commonalities and understand commonalities between different domains. On the other side, you sometimes need very specific knowledge in a specific domain to get the right prediction applied to this domain. For instance, if you only train an LLM on cooking recipes, it won't be really effective at creating some code or platform code. So those models are going to struggle a little bit more with underrepresented data. The issue is that with infrastructure as code, so Terraform in particular, the datasets are quite sparse. Why? Because all those amazing generative models that have been trained for the code generation parts have been mostly trained on GitHub. The issue is you don't put your infrastructure on clear on GitHub. It's very sensitive information, and so you don't have a lot of enterprise-grade information on GitHub. Also, it lacks the live context of the infrastructure, so all the run metrics and all the very specific production configuration metadata that you have. An example to illustrate that would be to look at the amount of code between Python to HCL, not even Terraform, on GitHub. You can see that you have like more than dozens of millions of Python files on GitHub. You only have two million on HCL files, which is even less for Terraform, so it's like a 32-factor between Python to HCL, not even Terraform. As a matter of fact, you have 1,500 public modules on GitHub, which is very low. So, what's going to happen because of that? First thing is that some of the tests we've conducted, even if it's improving a lot and every day, is that you can miss the right practices when you generate some code, infrastructure-as-code code. The prompt that we used was to generate a Terraform configuration for like a VPC peering connection. What you get here as a result is not incorrect. The VPC peering configuration worked. The thing is, it missed what an expert or someone with like some knowledge about how to do things well would have integrated. So, you miss the auto accept and tags variable, and also, you have some hard-coded values, which will cause an issue in the later stage of your infrastructure that can break, actually, your pipelines. So, here, the right configuration with the tags and auto accept. And the worst part is actually the hard-coded values. So, the dependencies here are missing. And what you should see would be normally to have some reference to the right resources defined within your codebase, but one of the biggest short-common challenges that those models have is that they will not have the context of your infrastructure at hand. They will not know where to fetch for the right resource and create some viable dependency. This is one of the short-common that you have when your model, which is a probabilistic one, will actually just generate something based on what has been trained on, but not on your specific infrastructure. Last but not least, in terms of limitation of your infrastructure as code generation, would be in terms of security issue. Here, I have taken quite a very big example, which is not that likely to happen anymore. So, having a model generating some open port, so, side of block equals zero, which is a very bad practice. It's still possible to happen. How come? Imagine you have 1,500 public modules on GitHub. You have some attacker that is going to actually create 200 new ones. Your next generation of models are going to be retrained on GitHub and are going to be actually trained on those new modules with those bad configurations. Because of that, they are very likely to generate this type of content, which is highly critical in terms of security. Because of this issue and the probabilistic nature of neural network and those LLMs, they will predict next code tokens based on probability, but never with 100% certainty. It's highly recommended to use deterministic tools, so, tools that will always give you the same output if you give the same input, such as like Chekhov or Snyk. A second example and security risk, which is even worse, with those LLMs, would be with the import of malicious provider. There's an amazing article on Snyk blog about that. By the fact that, imagine you have this attacker who has coded 200 new modules on GitHub. This person could have integrated into these modules a malicious provider. Because of that, and because your model are going to be trained on those new modules, you could be subject to attacks where you import into your code the wrong provider, a malicious one, which can lead to a leak of your AWS or cloud provider's credentials. So, that's the first part about how LLMs are being trained and how, actually, they still have some limitation when it is applied to the world of DevOps. Now, just to take one step back, it's also interesting to see that AI is a big world with a lot of concepts within, and so, there is an important distinction between generative versus synthesis AI. Amazing article from A16Z between the difference between the two. Both domains use LLMs, large-range models, AI models, to actually exist, but they won't incorporate information the same way. With generative AI, you give a prompt, so, some context as input, and you want new context to be created, new information to be created to be created from that. So, it's like an open-ended kind of game. You give something as input, and you want to create something new. Synthesis AI, on the contrary, is where you give a lot of information, but you don't want to create anything new. You want to find something within this information. You want to synthesise it. So, in the first case, you have low value of information as input, and you want to create a lot of information as output, and, in the other case, you have a lot of information as input, and a very small output. Because of that, synthesis AI tends to have better accuracy and results than generative AI for now, because the solution space is way smaller. You give a lot of information into the input, but the solution is contained within, and you just need to find it. The first application for that would be log analysis. You analyse a lot of information, hundreds of thousands, millions of logs, but you will be able to find the needle in the AI stack because those AI models are very good at finding patterns. So, even better, so, for instance, you have like Google Cloud Ops, AI, is to use that for root cause analysis. Generative AI is going to be amazing, sorry, synthesis AI is going to be amazing at finding patterns between heterogeneous sources of data. For instance, imagine you have a customer alert on your application, and you want to connect this information with your logs, and so you will take heterogeneous sources of inputs of failure, and you're going to be able to correlate and find back the similarity between those two alerts, thanks to those models that are very good at finding patterns. But, so, now that we spoke about synthesis AI and why it's amazing, the last part of this talk is about how can we actually still have some hope for a better generation of code, infrastructure code, platform code, for your infrastructure. This part is about how the solution resides in a context-based infrastructure, so those LLMs, those models, can actually generate some code based on your specific knowledge. To go back on the example of dependencies, if your LLM had the entire context of your infrastructure, it would have been able to generate the code with the right dependency. So, the main point in this part is that the future resides in a two-pillar kind of way of seeing things. You have the AI part for generating some code, but you also have the deterministic part which is a context part of things to actually generate the right code adapted to your infrastructure. First of all, let's speak about FWAG, which are retrieval augmented generation technology, and how actually this technology enables to have best of both worlds between generative AI and context. So, how does FWAG work is based on the fact that you have an LLM, you know that it's going to give too many broad answers to your question, and so you want to restrict it to your own knowledge. So, what you're going to do is that you're going to take all your company knowledge, so documents, JSON files, documents, code base, that you are actually going to pre-process. So, you are going to embed this data with this technology into a smaller chunk of data that you're going to encode, so you can see it here, into this mathematical representation, into this latent space. And then, based on that, once you actually store this latent space representation of your knowledge, of your company knowledge, you will be able, with a model, an AI model, to query this knowledge, and to be able to answer to some questions such as, what is the size of my company? And so, instead of having a general model that doesn't have a clue about who your company is, when you say, when you refer to my company, this model is going to only query this representation, like your knowledge, and be able to actually take back the specific documents that speak about the size of your company. The thing is, infrastructure is a graph. So, there's a slight difference between a document, which is a file, and a graph. So, you can see here that the way this graph is created is that you have a VPC, and within your subnet, and IAMs attached to it, and so the way that you create the context of your infrastructure is actually through a graph-based approach. So, you will get a slight difference between a baseline, a classical RAG, which actually encodes a flat list of documents, and a graph RAG, which is actually going to encode a graph of interconnected nodes. The way the baseline RAG is going to be searched will be based on similarity between the vectors that I encoded, whereas for the graph RAG, the query from a question is going to be based on the relationship between the different nodes. The core principle of a graph RAG is first to create this graph, which is based off nodes, so resources, VPC, subnets, everything that lies into your infrastructure, and the edges, that has a relationship that ties those nodes together. To actually leverage this graph, at query time, when you have a question, you're actually going to go through this entire graph to actually ask what are the connections between those different nodes. The core challenges in graph RAG is that the core challenges in graph RAG is that first, its construction is kind of complex, and second, is that the traversal is kind of complicated. So, first of all, the construction of the graph is kind of complex, because all the complexity, the difficulty, is in the processing of this knowledge. The way you're going to structure the relationship between the nodes is going to create a different quality for your context. It's not the same way to actually create a connection between a VPC and its subnets than to actually create a connection between similar tags. It's not going to actually create the same kind of context within your infrastructure. So, this is very domain-specific meaning, and like work, to define the different edges, and what does it mean. And then, how to query from that, on real time, this graph to actually being able to get the right answer to a question, and how to actually embody, how to encode this information. So, as conclusion, through, like, so different topics, Gen AI applied to DevOps and infrastructure is amazing. It has made already amazing things, and has amazing, like, promises, but still have some limitation. One of the biggest bets to actually, like, create the right content for the right information is the context of the infrastructure which is available, and for that, because the infrastructure is a graph, it's important to actually create this context as a graph. And all the challenges in the future would be how to actually, like, embed this graph in the right manner, and being able to query it in the right manner. Thank you very much, and very happy to have presented this talk today. Thanks.
