Transcript
Hi, my name is João Pita Costa, and I am Senior Technologist in Artificial Intelligence at OpenEBOLA. To share the experience on how OpenEBOLA features are supporting AI workloads, today we have an excellent opportunity to talk with Jean-Philippe Fauré, VP Product for the French number one cloud and DevOps experts Iguana Solutions, and Kim Henriksen, SVP for Technology, Innovation and Ecosystems at the Swedish National Centre for Applied AI, AI Sweden. Iguana Solutions are building AI platforms for their customers using the OpenEBOLA flexibility to unify workloads in the same private cloud, supporting GPUs by utilizing the GPU passthrough and using one to provide a private cloud to DevOps teams moving onwards to build LLM layers and core services. AI Sweden is using OpenEBOLA to manage on-prem infrastructure spread over three different locations, providing a testbed for AI experimentation, learning, and collaboration for over public and private institutions, hosting a variety of projects, frameworks, and workloads, and leveraging a direct connection to GPUs, CPUs, and data. OpenEBOLA enhances the orchestration capabilities based on tailored configurations to deploy AI processing clusters, ensuring cloud edge location to run AI workloads. We care about latest AI applications where it is crucial to place the execution of AI and ML pipelines closer to the data sources and reduce the data transfer times and bandwidth usage. Taking LLM AI appliances deployed as services on VMs with metrics integrated to our monitoring system, we are improving scheduling and workload optimization. The servers include specialized hardware like GPUs and TPUs that benefit AI workloads as AI workflow tasks require high-performance CPUs and high-capacity RAM to fit models in memory. Apart from this high-end hardware setup, the confirmation of the server also requires the VM to direct access to the GPU cards or to enable the use of virtual GPUs with an orchestration that can properly manage the allocation of PCI devices to VMs, control resource usage, and account for or even limit the usage of these special devices. We are preparing a framework to contribution discovery and usage of applied applications tailored to run efficiently on top of an OpenEBOLA cloud infrastructure based on our appliance marketplace, including certification and maintenance processes, and ensuring that appliance remains compatible and up-to-date. The realization of an effective cloud for AI models requires an implementation-as-a-service paradigm where users can essentially LLM virtual servers with pre-trained models for different use cases. The OpenEBOLA user can count with a range of performance features through the enhanced platform awareness, supporting the extensive collaborative work with customers and partners. This platform enables the fine-grain matching of processor capabilities to VMs and Kubernetes workflows prior to launching the applications. It allows to improve the VM packet forwarding performance and provides GPU support, bare metal automation, and multi-cluster features. In particular, the PCI pass-through allows to discover PCI devices in hosts and directly assign them to virtual machines. In the KVM, I provide allowing direct access to GPU hardware by virtual machines and providing the necessary computational power and performance needed for intensive AI applications. By supporting NVIDIA vGPU, OpenEBOLA facilitates the effective distribution of GPU resources among several virtual machines, improving computational performance, decreasing latency, and optimizing resource usage for detailed and complex deep learning operations. And I will now leave you with the distinguished guests, starting with Jean-Philippe Fauché at Iguana Solutions, and then later on followed by Kim Eriksson at AI Sweden that will share their experience and advice in using OpenEBOLA in their AI-focused use cases. Thank you, Joao. Hello, everyone. I'm happy to be here to talk about AI today. It's changed me because in the previous conference, I talked a lot about Terraform provider for OpenEBOLA. So, at Iguana Solutions, I don't know if you followed the previous session with my CRO, Cédric Lemoine. At Iguana Solutions, we manage platforms for our customers, and we use OpenEBOLA for more than six years now in production. Today, I am here to explain how we use OpenEBOLA to run AI on our clouds. So, let me start with the today agenda. So, basically, it will be very simple. I will explain what we do with OpenEBOLA, then what we can improve, what we are currently doing with OpenEBOLA, and then a small conclusion. So, today, where are we? Today, everyone is aware of AI. I know you listened to this word for a long time since GPT released its first release two years ago. So, I will use the AI terms a lot in the next 10 minutes. So, for me, contrary to the word tree or crypto, AI is not a hype. I think I'm convinced that we are at the beginning of a new era, and we are convinced that AI will help a lot of people in doing better their job. It won't replace them. It will help to be better. How? So, today, the market is very huge. Actually, we estimate that we are starting about $600 billion, and we will be multiplying by three in six years. So, we want to be part of it as platform management. And today, AI, for example, is used a lot by developers. Ninety-nine developers on 10 use AI-based tools, and they use, essentially, GitHub Copilot. I don't know if you know this tool, but it's amazing for developers. And 70% of them say that it adds them significant benefits for their usage, and it improves them doing better code. It's crazy. But all organizations can't go on public cloud, and all organizations can't use on-demand AIs. That's the case we want. That's the problem we want to solve at Equant Solutions. So, our mission is to help organizations benefit from AI by providing them AI platforms. So, what's an AI platform? And we want to use OpenNebula as a pillar of our AI platform. Because OpenNebula offers flexibility and the capacity to run VM containers, private cloud integration. In our case, we do private clouds. It means that we can make virtual machines on OpenNebula clouds and bare-metal servers in our data centers. It also provides unified workloads on the same private cloud, and also, as Joao said in his introduction, there is a GPU support thanks to the GPU back-through mode. So, an AI platform based on OpenNebula has several layers. We start at the bottom with the infrastructure layers, with servers, with GPU servers, and we will have GPU hosts with NVIDIA H100. And in the coming weeks, we will also have the H200 GPUs. StorePool is a very good provider for storage. It's very performant, and I invite you to follow the StorePool sessions, too, because it's an amazing solution for storage as OpenNebula. And VXLAN network for the data center virtualization. So, on top of this infrastructure layer, we add OpenNebula to manage our virtual clouds with GPU support, obviously, thanks to the GPU back-through and CPU pinning to be sure that the CPU cores of the server are entirely dedicated to the VM. On top of this cloud layer, we add several services to manage the different models. So, I will call this layer the LLM core services. There is a specific layer with LLM servers to run the different models, to load the model on the GPU, to offer an API to dialogue with LLM. And on top of these LLM servers, we add an LLM proxy to redirect, to manage the API keys, to manage the load between the different servers, and also to instrumentate the observability, because it's very important to, as a platform, to be able to monitor it and to follow the LLM usage. So, this observability layer is connected to our own observability stack. It's a homemade solution based on Prometheus for metrics collation, Grafana for data visualization, and Victoria Metrics for long-term storage of the metrics. So, these three layers, hardware, OpenNebula, LLM core services, is our platform, and on top of it, we can plug any application to run AI workloads. So, we already used internally a chat, a co-pilot, a private co-pilot. We also have a web search tool, a no-code application development tool. There are lots of amazing applications. We use it internally at CoinSolution. And obviously, we present an API compatible with OpenAI. Thanks to this API, anyone who already started developing applications using the OpenAI API can just change one line of its code to use this private platform. And so, I leak the usage. In our use case, we use it as a private co-pilot for our developers. It's crazy, because this private co-pilot runs on a private platform, so it means that there is no leak of intellectual property, no risk of that. Obviously, there is no risk of data privacy breach. And this co-pilot scans every time your baseline, your code base, so suggestions are doing better and better. And also, you can upload any code documentation and interact with this co-pilot to ask him how to code specific functions on a specific language, and the AI will answer you. Obviously, as a platform management company, we have to provide a lot of documentation for each of our customers, and using the AI to maintain the documentation will speed up the process and will increase the quality of our documentation for our customers too. And our project manager, our CSM, will love what we are thinking about to help them. It's the automatic report. Every month, we can generate automatic reports of the platform usage, the project usage of each of our customers, and we will enrich with AI to provide a good feedback on monthly usage. And it will be easier for our team to write such documents thanks to the AI, and it will speed up the process, and it will take them a few minutes to rewrite some sentences instead of taking hours to do each report. It's crazy. So, that's our current usage. But what we can do more with Open Nebula? What can we do more? So, what's next? And we are working with Tito and his team to add more integration, native integration in Open Nebula. Okay, I need to speed up. So, we want to increase, to add in Open Nebula appliances, and an equivalent of NVIDIA K8S device plugin, different appliances natively integrated in Open Nebula, and also the metrics in the same dashboard. So, based on our AI platform, anyone who wants private AI stack, it's a plug-and-play platform, infrastructure and analytics stack management is included, and also user-facing features are already part of this platform. If you want to know more about that, so, sorry, about the conclusion, Open Nebula and EGWEN story starts six years ago, and we believe in Open Nebula as a pillar of our AI platform, and we are working closely with Open Nebula to build the next generation of Open Nebula clouds. And for us, it's just the beginning. If you want to know more about what we are doing with our AI product, I invite you to scan this QR code, and it will be my pleasure to provide more details about the platform itself, the tools we are using, the stack we deployed, where we are deploying, and what kind of workloads we can do today, and what kind of workloads we are thinking about doing in the future. So, thank you for listening to me. I will let Joao and Kim manage the next session. Hi, everybody, and thank you, Jean-Philippe and Joao, for those interesting words. Actually, yesterday, I had a discussion about private clouds in Europe and finding suppliers of that, so I will definitely make them aware of EGWEN. So, I come from an organization called AI Sweden, and we're a national center for applied AI. Our purpose is to accelerate the use of AI for our society, our competitors, and everyone living in Sweden. The rough outline of my next couple of minutes here will be first to tell a little bit about who we are, then why we do what we do, and it will all be in an Open Nebula context, you will see. And then, of course, how, and that's where I think it gets interesting. You can see all of this as a little bit of a case study in how one organization utilizes Open Nebula, and we're pushing Open Nebula to its limit in some extent. So, applied AI in Sweden, and we're a neutral nonprofit organization that's broadly funded, and we are both for private sector, public sector, and academia at the same time. And here are a little bit of our partners. This goes from global tech down to startups, and then we have public sector, lots of government agencies and municipalities and cities in Sweden, and, of course, academia. Something that's not on this slide is our collaborations with MIT and Dakota State University and Cambridge and so on. But what we're trying to achieve is the collaboration between these parties, collaborating around how to implement AI and start getting value from AI, and doing their transformation journey. We have a saying, and that's invest together and share with many. And with that, a part of it, a really important part is speed, and how can we make it quicker for our partners to reach what I would say, let's call it organizational confidence, meaning that how can we give them the hands-on experience with certain tools that are out there. Whether it's either open source or proprietary, requires some certain infrastructure to set up to begin with, or a specific architecture, or similar to that. So, very much a compliment to Egon that we just heard about. And I belong to a part called AI Labs, where we try to give this hands-on access to infrastructure that runs AI. And this infrastructure, even though we want to do a lot of things with it, and it's open for partners and partner partners, it's a good question into how do we practically do this? And that comes down to our partners contributing hardware and software, and we run it in just one big, large, I would say infrastructure of infrastructures. This means that we need to have a way to handle multiple hardware vendors, different servers, and various type of edge nodes or edge IoT devices, also NVIDIA, AMD, Intel GPUs, and other accelerators. And that all combines to, we need some way to do virtual machines and container management and so on. And here is where OpenEVLA comes into the picture for us. And what we're trying to serve is really a testbed that can have its multifunction, multipurpose. So, the demands we're putting on OpenEVLA, in this case, are quite high in terms of flexibility and being able to do a lot of things. But as it turns out, if you can host virtual machines in various VLANs or, let's say, small private clouds, then it becomes quite easy to do so. We've also, and it's not part of this presentation really, but we've actually spread this out over three physical locations, which, if you think about it, having one data center spread out geographically and trying to administer it from one point is actually quite a challenge. But here again, we're actually leveraging OpenEVLA's pretty good, I would say, ability to manage VMs and workloads wherever they are. Now, let's get into the interesting part. And I've been teasing a little bit here. So, we have heterogeneous hardware. The testbed is run by, let's say, administrators that are both from our organization, but also our partners' organizations. So, it's all a big collaboration. So, that puts quite a lot of high requirements of it being easy to use, whatever it is. It should be easy to configure. It should be easy to set up. It should be easy to change. And the hardware and software comes and goes. And really, we use it as an abstraction layer between hardware and whatever software that is run, either AI workloads directly or other frameworks that just, and then that in turn, deliver something else. So, it's a constant motion here. And there's a small picture here showing a bunch of projects that are ongoing at this time. So, it creates a lot of complexity, and that we try to reduce using OpenEVLA. So, what we really like about OpenEVLA is it's how easy it is to add and remove hosts. Or that it's relatively easy. And I would say here it's especially flexible, and that's part of the learning as well. But templates, just having that be a no-brainer, it's not available for your OpenEVLA's competitors in that sense. You don't get really the same. The context management, being able to add SSH keys if you have SSH contextualization, being able to add passwords and settings to any of the machines live. Also, doing migrations, both live migration and regular migration, and resize machines and stuff like that. It's very intuitive, very easy to use, and we noticed that it saves us time. Then, of course, the piece de l'horizonte, GPU pass-through. Absolute must for any type of AI workloads. Works. No real questions there. I'll also note, as a sort of third part in this, that OpenEVLA is very price competitive, even as far as having a community edition that is open source. We actually go between using the open source and the enterprise version. Yes, and some critical learnings here. There are pros and cons. OpenEVLA is a bit of a David versus Goliath, but one of the main pros here is the flexibility that that creates, and that we can do more of what we want. I will say that's all for me, but if you want to know more about AI Sweden, check out our community at my.ai.se, and with that, I will hand it back to Francisco and Jao. Thank you very much. Thank you very much, Kim. We have time for one question or two. Maybe I would like to start with Jean-Philippe. What are the LLM usage metrics that should be prioritized and are more important for the business scenarios of your users? It depends on the users, actually. Good question. Actually, in terms of metrics, what we are thinking about, we want to provide usage metrics. On the dashboard, a customer using our platform will want to know which kind of models are used by people in his company or by his application. And the important metric is the success. I mean, if the result of the query is good for the user or not. Does he love it or not? Because it will help to increase if the model is good or if it's not a model. And then, I think, as a user, it's this kind of metrics I expect. And we want also to add the system metrics. Is the platform loaded or not? To know if we need to add more resources to expand its usage. Thank you. And one question now for Kim. In your opinion, what are the bigger challenges in AI research and industry in Sweden? And how will Open Ebola empower AI Sweden to address them? Good question. I would say the sovereignty, both from a Swedish perspective, but also European perspective. And I guess, Sean, Filip, and you guys have also seen that we want to host our own models. We want to secure our data. We don't want to know where it is. And doing that requires that you either have your own on-premises infrastructure, or you have a trusted cloud provider that is Europeanly owned and so on. And when you're doing that, you need some way to control your resources. And that's where I see Open Ebola comes in and also AI Sweden. Thank you a lot to both of you. It was a very exciting panel. I will now pass on the ball for the new sessions to come. But surely, I invite everybody that is further interested to interact with Filip and with Kim over there, over their channels and over the solutions in the AI Sweden. Have a nice day. Have a nice day. Thank you. Bye.
