Transcript
Hello, and good morning, good afternoon, or good evening, depending on where and when this finds you. Thank you very much for spending the next while with me. My name is Steffen Bischof, I am Chief Architect Data at Qlik, and today I want to walk you through agentic data engineering. What it is, why it matters, and where we are taking it. One warning before we start. This is not a five-minute highlight reel. We are going to go deep. We are going to spend most of our time in a live demo, and at some point, there will be actual YAML on the screen. If that is not your idea of a good afternoon, now is the moment to close the tent. If it is, welcome. You are among friends. And everything I show you today is running software. It is not a mockup. And it is not a roadmap slide with a nice gradient on it. Wherever something is still ahead of us, I will say so clearly. Here's the plan for today. We start with Qlik Talent Cloud's declarative data pipelines, our new way of building enterprise pipelines, including how AI tools like Virtual Studio Code, GitHub Copilot, and Cloud Code can generate them for you. From there, we'll look at the agents we've built for Qlik Talent Cloud itself, data quality, data products, catalog and glossary, help, and analytics automation. And at the end, I will show you how to take the AI tool of your choice and connect it to all of that over MCP. Roughly a third of this session is slides, and two thirds is hands-on keyboard, which is the ratio I wish every vendor presentation had. One more piece of setup before we begin. Rather than talk in abstractions, I want us to follow one organization through the whole session. Over the past year, I have worked with several insurers on exactly this problem, and instead of telling you five half-stories, I have folded them into one. I will call them Meridian. Meridian is a mid-sized insurer, around 2,000 employees, a core policy system nobody wants to touch, and a data team of six people who are very good and very tired. Their data is roughly what you will see in the demo, and none of this is insurance-specific. If you are in retail, manufacturing, or the public sector, swept claims for orders or shipments or cases, and every single point still lands. The way data pipelines get built is changing. And most teams are feeling it from the wrong end first. Your engineers spend their week on plumbing while the request queue for data products keeps growing. At Meridian, the number that made this concrete was 11 weeks. That was the average time between a business team asking for a new dataset and that dataset running in production. Not because anyone was slow. 11 weeks of connectors, schema mapping, review cycles, environment promotion, and waiting for the one person who knows how the policy system encodes cancellations. Enchantic AI changes that equation. But only if three things come together. First, intent-driven design. You describe the outcome you need, and the platform generates the pipeline. Two effects for your team. People who know the business but not the tooling can contribute. And your senior engineers stop writing boilerplate. The interesting number is not lines of code saved. It is how long a new request sits in the backlog before it runs in production. When 11 weeks becomes 11 days, the conversation with your business stakeholders changes character entirely. They stop asking for everything at once, because they no longer have to. Second, autonomous operation. Once a pipeline is live, the platform uses its own metadata to see what is happening. Schema drift upstream, loads that failed, workloads consuming more compute than the result justifies. Let me be concrete about why that matters. At one customer, a supplier changed a field from a two-character country code to a full country name. No announcement, no ticket, just a Tuesday. The pipeline kept running. It ran beautifully. It produced a regional report in which 40% of the business had quietly relocated to a bucket called Other. That was discovered the following Monday, by someone in Finance, in a board pack. So, the point is not reducing headcount. It is how much change your team can absorb without adding people, and how many Monday mornings you get back. Third, trusted, AI-ready data. An agent is only as good as what it reads. Without lineage, quality rules and access control in place, you get answers that sound confident and are wrong. And that is the failure mode that kills AI initiatives right after the pilot. I have watched exactly that happen. An internal assistant, six weeks of enthusiasm, and then somebody asked it for the average claim value. It answered from a test table a contractor had left behind two years earlier. The number was entirely plausible. It was also off by a factor of three. The assistant was switched off eight days later. And the real casualty was not the tool, it was the appetite to try again. Governance is what makes the output defensible to your risk and audit people, and defensible is what keeps a project alive past the pilot. These three only work together. Fast pipeline creation without governance produces mess faster. Governance without automation is roughly what you have today. So here is a question worth sitting with before we get into the product. Which of those two hurts more in your environment right now? The speed at which you can build, or the trust in what is already running? Your answer tends to decide which half of this session you will care about more. Let's get concrete about what declarative data pipelines actually are in ClickTown Cloud. The first thing to say is the thing nobody says often enough. This is an additional way to build pipelines. It doesn't replace what you already have, it adds a new option. Nothing you have built stops working, nothing needs migrating, and there is no deadline hanging over you. I say that deliberately, because most people watching have lived through at least one exciting new paradigm that turned out to be a two-year migration project wearing a nice jacket. This is not that. You get to choose your development approach. Visually, programmatically, or agentically. Meaning you describe your intent, and the agent builds it. Different people on the same team can choose differently and still work on the same pipeline. Your visual builder and your YAML person are not in a turf war. They are looking at the same object from two sides. You also get to choose your own tools. Whatever agent you prefer, whatever IDE you are already working in. And critically, this is integrated into the existing architecture. It's a perfect complement to our APIs, and it's natively integrated with version control. In the following example, we are showing Virtual Studio Code as the IDE and Cloud Code as the coding agent. But neither of those is a requirement. What do you actually gain from this? Three things. First, automation. You build and automate your pipelines programmatically, using a simple YAML coding experience. YAML being the industry standard your teams already know, so the configs are easy to read and edit. That readability matters more than it sounds. If an agent writes something you disagree with, you can simply fix it by hand. You are never locked out of your own pipeline. Second, build once and repeat. You design the pipeline in YAML a single time, and then you deploy it across development, QA, and production, without repeating manual steps in the UI. Meridian had a Word document for this. 41 screenshots, titled Pipeline Promotion Procedure Version 7 Final. Every one of you knows that document. It lived on a shred drive, it was permanently two versions behind the actual user interface, and the colleague who maintained it had left the company in spring. Promotion took a morning and a certain amount of prayer. Now it is a file in Git and a pipeline RA. Third, the ICD integration. Because the pipeline lives as code, it becomes pivotal to your development workflow rather than something bolted on beside it. It sits in Git, so you get proper versioning, pool requests, code review, and full traceability of every change. When somebody asks in six months why a transformation looks the way it does, there is an answer with a name and a date attached to it. So if you remember one thing from this slide, automate with simple YAML, reuse across environments, and embed it in your CICD for versioning and collaboration. All right, that was a lot of theory, and theory is where demos go to be quietly replaced by screenshots. So at this point, I'd like to stop talking and show you. Let's switch over to a live demo. What you are looking at is a pipeline I prepared earlier, and which is already running. The data belongs to Meridian, our insurer. As I said, the data itself could come from any industry and any use case. The point is simply to use an example concrete enough to be useful, one you can map onto your own world. We see three very simple steps in our pipeline. We connect to our core system, which is based on MySQL. And we replicate every change directly into our Snowflake database. That happens fully automatically, using our change data capture technology. We read the MySQL system log to detect changes, and we replicate them into the target system in real time. And it is worth pausing on what that avoids. Nobody is querying the production policy system to build a report. The core system barely notices we are here, and the DBA who guards it can carry on sleeping. In our example, the source is MySQL and the target is Snowflake. But of course, a long list of other systems is supported out of the box. After the landing task, the storage task takes over and makes sure the data is already prepared, analysis-ready. This is where transformations can happen, and where the data is historized automatically. For all the nerds among you, and I count myself firmly in that group, we take care of slowly changing dimensions type 2 for you. If you have ever written that logic by hand, you know it is one of those problems that looks like an afternoon and turns into a fortnight. At the moment, there are no transformations. And the storage task is not yet scheduled to process the data periodically. That is exactly what we are going to change, and I'm not going to write a single line of YAML to do it. Our pipeline is already connected to Git, which means it is stored in a central code repository. So, we essentially treat our pipeline as code. We can pull changes down from the repository or commit our own changes up into it. That is what makes the next part possible. It allows us to work on the pipeline as code in a completely different environment. In my case, in Visual Studio Code, with the help of Cloud Code as our AI assistant. So, let me open VS Code. At the moment, my directory contains exactly one file, the Cloud MD file, which tells Cloud Code how it is expected to behave. In our case, we point out in no uncertain terms that nothing may be changed without reading the rules of engagement first. Let's open it and check. You can see it describes precisely how a pipeline is structured, what it consists of, and so on. Think of this as the onboarding document you always meant to write for new team members. The difference is that this particular colleague actually reads it every single time and does not skim ahead to the section with the connection strings. Now we clone the repository containing our pipeline so we can carry on working with it here in VS Code. A single click in the user interface is enough, or we use a simple git command if we want to feel like professionals. From here, I want to make changes to our pipeline in natural language, with the help of Cloud Code. To be clear about what is happening, I do not want to write the changes in YAML myself. I describe my intention and Cloud Code implements it. Let's start by defining a schedule. Specifically, the storage task should run automatically once a day, starting today. Cloud understands my request and starts editing the files accordingly. Notice that it isn't guessing at the structure. It knows the structure because it read the rules first. That looks good. So let's move on to the next change. So let's move on to the next step. In Snowflake, the names of tables and columns are stored in uppercase by default. Rather than remembering that every time, a simple transformation rule should take that logic over for us. Here too, Cloud Code understands my intention, knows the rules from our Cloud ND file and takes care of the implementation. So far, so mechanical. Let's try it with slightly more complex rules. I want to add a new field that calculates the processing time of an insurance claim, so how long it took to settle the case. And there is a catch here, the kind of catch that makes real-world data interesting. We have to account for the fact that some cases have not been closed at all yet. In that situation, I still want to record how long the case has been in progress so far. That is a small requirement with a gene-wine trap inside it. Handle it naively and every open claim gets a null, or worse, a processing time of minus 50 years because something got subtracted from an empty date. Let's see what we get. Wonderful. And now, a categorization of insurance premiums. What I want is a grouping of premiums into different categories. Minimum, maximum and three further categories, where P50 represents the current median of the data. For my test, that is exactly the right fit. Okay, those changes have been made as well. Now I want to mirror them back into Git and update my pipeline from there. This time, I'll use the VS Code interface directly. Write a commit message for traceability, because future you will absolutely want to know why this happened. And click commit and push. In this case, I am simply trusting that it all worked. In a real project, you would not have to. We also provide an API that would let me check right now whether the pipeline is still valid and whether the YAML files still conform to the conventions. That is exactly the kind of check you would wire into a pull request. We'll spare ourselves that step here. Back in Tenant Cloud, we now synchronize the changes from the Git repository into the pipeline. And we can trace precisely what was changed, line by line, before anything happens. This is the part I would draw a box around if I were taking notes. The agent proposed, Git recorded it, and a human approves it. Nobody is being asked to trust a black box. One click and we accept it. After a refresh of the pipeline, let's walk through the changes we made together. We can see that the storage task is now scheduled and will run once a day from here on. And the storage task now contains a whole set of transformation routes. All tables and columns are written in uppercase. Snowflake is happy and nobody had to remember anything. In the claims table, we can see the calculation of the processing time. Now I don't want you to take Claude's word for it, and frankly neither should you. To check that Claude formulated the expression correctly, we can enter sample values and simply test it. Let's try a closed case. Opened on the 20th of January 2025 and closed on the 1st of January 2026. And if we take the same case and set it to open, we see the number of days up to today's date instead. Wonderful. Which also means an open claim now tells you how long it has been sitting there. If you have ever wanted to find the claims that have been open long enough to develop opinions, this is the field that finds them. And in policies, we can follow the grouping into the different bands. We can test that easily by entering a value of 3000. Or a value of 100,000, which should land in the highest category. So, four meaningful changes to a production-shaped pipeline, expressed in plain language, reviewed as code, versioned in Git, and deployed. And I never opened a YAML file. There is an important option in the storage task that I want to switch on now. It ensures that all records are not only processed in Snowflake, but are also published directly into ClickCloud as datasets. That is the bridge between the first half of this session and the second. Once the data is available as a governed dataset, we can carry on working with it manually and, far more interestingly, with the help of our agents. So, let's switch over to ClickAnswers, our agent-in-chief. All the specialized agents gather underneath it, and we can call them from here. You don't have to know which one you need. I have a personal habit whenever I meet a new agent, which is to start with the simplest possible question. What can you do? It's the same question I ask a new colleague. Except with colleagues I usually have the decency to wait until the second week. Okay, that sounds promising already. Let's find out which data products even exist and which ones I am allowed to access. Note the second half of that sentence. I'm not seeing everything that exists. I'm seeing what my permissions allow. Let's look at the Northwind data product a little more closely. It has nothing at all to do with our insurance data, but I'm curious anyway. And if you have worked in this industry for more than a week, you have met Northwind before. It is the demo dataset that simply refuses to die. Okay, confirm. Genuinely, nothing to do with our insurance data. So I'm going to have to create a new data product. Let's see whether the agent can help me with that. Naturally, the agent needs a few more pieces of information. And it asks for them in a very structured way. Of course, I could equally have left the name, description, and tags empty and asked the agent to propose them itself later on. I'm filling them in because I happen to have opinions, but the option is there and it is the one I would use if I were creating 20 of these. Either way, we are already a good step further. Now I select the individual datasets that should become part of my data product. There we go. Okay, so we have the first version. Let's ask our agent what else we could do to improve the quality of the data product. This is the part I find genuinely useful. It isn't just doing what I asked, it is telling me what I forgot. Let's take a look at our new data product in the data products catalog. ClickAnswers is reachable here too, so we can use the data products agent right where the data product lives to understand it better. So, from our data in the MySQL database, we have already produced a finished data product. And I want to be precise about where things physically live, because this question comes up in every architectural review. The data itself is still sitting in the Snowflake environment. Only the metadata, the descriptions, the data quality metrics, and so on are in ClickCloud. We did not copy your data somewhere else in order to describe. Speaking of data quality, I'm fairly certain we should take another look at that. Let's examine one specific dataset, claims, in a bit more detail. Unfortunately, there is no description for the dataset yet, which is the normal state of the world. Descriptions are everybody's second priority, permanently. But here too, an agent can help us out. It is remarkably easy to have the description generated automatically. And not just for the dataset, but for every single field. That fits perfectly. Now let's get properly acquainted with the data quality agent. And as always, I want to meet it first and find out what it can actually do for us. Every agent also offers a selection of ready-made questions or tasks in the initial conversation. In this case, I want to use that option and ask for a short report on the data quality of this dataset. Okay, so there is still a bit of work left for us to do. And here too, I want to fall back on the help of the data quality agent. Let's look at the data itself in the data preview for a moment. For every column, we can see not only the technical data type, such as string, integer and so on, but also what we call a semantic data type. In other words, what the data actually means. To do that, we use word lists or patterns or a combination of both. A large number of pre-built semantic types already ship with the product, and they can of course be extended with YORO. That distinction is more useful than it first sounds. A technical type tells you a column contains an integer. A semantic type tells you it contains a postcode. And postcodes have rules that integers do not. In a new chat, I want to create a new data quality rule. Specifically, whenever an error flag is present, meaning it is set to 1, a reason has to be given as well. This is one of my favorite categories of data problem, incidentally. An error flag with no reason attached is the data equivalent of a colleague shrugging at you. The agent proposes the rule and the matching fields for it. We only have to confirm, and then the data quality is recalculated and the view refreshes. That was so easy that I feel like creating a second rule. This time, I want the report date to be after the incident date. Logically obvious, but important to actually check in the data. Because if it isn't true, you either have a data quality problem or a clean-winely remarkable claims department. Okay, that worked wonderfully too. Two rules, no rule editor, no syntax to look up. Let's also take a look at the lineage of this dataset. We can trace exactly where the data came from and what happened along the way, before the dataset became part of our data product. And this is the answer to the question that ends most data conversations badly. Where did this number come from? Here it is a click, rather than an archaeology project. Thanks to the data product, we now have a rather lovely overview of our data. Descriptions, tagging, data quality, lineage, and an impact analysis. But one important thing is still missing. Something that would make it considerably easier to understand which data we actually have and, above all, what it means. Precisely, the business glossary. Let me tell you why I care about this more than the slide implies. At Meridian, the phrase closed claim meant three different things in three different departments. Operations meant the file was closed. Finance meant the final payment had cleared. Legal meant the appeal window had expired. Three teams, three numbers, all correct and all different, and one recurring meeting to reconcile them. That meeting had been running monthly for two years. Nobody had ever written the definitions down anywhere a machine could read. And here, too, an agent can help us. We start with a familiar question to get to know each other a little better. And now we use the business glossary agent to create a business glossary based on our data product. So, the demo doesn't take all afternoon. We limit it straight away to four categories and a maximum of 10 business terms in total. Here, too, the agent needs a bit of information from us. We want to create the glossary in my personal space and we'd like to link it to the associated data product. We can also have semantic and business questions answered. For example, how the policy management That is the agent reasoning about the relationship between two parts of the business. In the glossary, we can see the linked data products here. Unsurprisingly, only our own, the one we created a moment ago. Perhaps we can improve the description a little. Let's simply ask our agent. Very nice. I'll take that straight into my glossary. And notice what just happened. The definition now lives in one place, attached to the data, versioned and visible to everyone, which is roughly the opposite of a monthly reconciliation meeting. Everything we have just seen with the agent can also be done outside of Qlik. That is what our MCP server is for. It provides a direct connection for the AI tool of your choice. In my case, I'm using Claude. But of course, our MCP server works just as well with Copilot, ChatGPT, Gemini, and anything else that speaks the protocol. We connect to Qlik Cloud initially, using my own user account. So Claude does not have any more permissions here than I would normally have. It uses my user context for every question. Which is either reassuring or mildly worrying, depending on how you feel about my permissions. But the principle is what matters. The agent inherits your access model. It does not route around it. And this is where a second kind of value unfolds. We now combine the reliable, curated, trustworthy data from Qlik with the intelligence of Claude. And that combination is the actual point. AI on its own is confident. Governed data on its own is inert. Together, they are genuinely useful. Let's ask what Claude knows about data products that have something to do with insurance. Wonderful. So it found the data product we created earlier and can tell us a bit about it. And it is immediately critical, thinking along with us, making suggestions for improvement and stating openly what it has noticed. I never asked it to review anything. It reviewed anyway. As I mentioned, Claude brings the data product we created earlier and states openly what it has noticed. There is a great deal of knowledge that goes well beyond our own data. Regulatory requirements, GDPR, and so on. So let's put that knowledge to work and find out which fields in our data are particularly worth protecting. And we may as well have it produce an HTML report, so we can circulate the result internally straight away. Think about what that combination actually is. The classification knowledge comes from the model. The field-level detail comes from our catalogue. Neither half would have produced this on its own, and the whole thing took about as long as it takes me to describe it. We can also produce data analysis with the help of Claude. And once again, because it is the single most important sentence in this demo. These are the data we saw earlier, the data we trust completely. These are not hallucinated numbers from Claude that merely look plausible. These are all well and good. But I would like a complete dashboard with the important KPIs based on our data. And Claude understands that it is connected to Qlik. So it starts entirely autonomously to build a native Qlik dashboard through our MCP server. Admittedly, it is not especially beautiful. It looks like something a very capable person built at two in the morning. But it is a real Qlik dashboard with all the functionality a Qlik dashboard offers. It sits inside the platform where your governance already lives. And it took under a minute. Beauty is the second pass. Existence is the hard part. Wow, that was quite a lot for one day. Wow, that was quite a lot for one demo. Let's step back out together, look at it as a whole, and round it up. What you saw in the demo was only the first step. This slide shows where it goes from there. On the left side is the data engineer. He keeps working where he already works, in Visual Studio Code for example. The coding agent there ships with an MCP client, and that client talks to the MCP server in Qlik Cloud. The server exposes two things, tools, meaning the actions the agent is allowed to perform, and resources, meaning the context it is allowed to read. Behind that sit the Qlik Talent Cloud Pipelines APIs, secured through OAuth. So the agent gets no special privileges. It runs under the user's identity and through the same permissions as any other access path. There is no service account holding the keys to everything, which is usually the moment in an architectural review when the security team stops looking for the exit. The box in the bottom left is the one that comes up most often in customer conversations. AI inference. You can run the model in the cloud or on-prem, depending on what your compliance requires. I have had this conversation in every regulated industry I have worked in, and it is almost always the first question and almost never the last one. Next to it is the AI instructions file. That is where you put your own conventions, naming rules, standards, project context. It looks like the least interesting box on this slide, and it is the one that decides whether this works for you or not. Here is why. Before Meridian had one, two engineers asked their agents for essentially the same transformation and got two different Yarnel styles back. Both valid, neither matching the other. That is not an AI problem. By the way, it is exactly what happens with two human contractors and no style guide. Once they had written the instructions file, both agents produced the same shape. That file is the difference between reproducible results and one-off lucky hits, and it takes an afternoon to write. Bottom right, Git, in two rows. The public QTCP repository provides schemas and scripts. Your own protract repository holds the pipeline Yarnel. The dotted arrows are formally optional. In practice, I would put it differently. If you are serious about this, you take that path. Git is what makes pipelines versioned, reviewable and deployable through the CI-CD you already have. An agent without version control is simply a very fast way to make changes nobody can review. Which brings me to the outlook, and at this point I am leaving the slide. Today we are talking about code. The next step is the pipeline itself. There will be two ways to get there. Either through MCP, with whichever AI assistant you already use, or through a specialized pipeline agent built for exactly that job. The point is not the agent. The point is that the interface stays open, so we do not have to commit to our assistant for this to work. Let's recap the second block. Three specialized agents, and you reached all of them from one place. Click Answers. That is the part worth holding on to. Nobody had to know which agent was going to answer the question. Which matters, because the people who most need these agents are precisely the ones least likely to know the org chart of our tooling. Quick recap of what you just saw. The data product agent took a plain language request and produced a governed data product. Create, maintain, and govern through natural language. The value is not that someone saved a few clicks. It is that the trusted product becomes the easy path. And people take the easy path. That is the whole mechanism. Every shadow copy in your organization exists because at some point the governed route was slower than exporting to a spreadsheet. Then the data quality agent assessed the dataset and proposed the rules. Quality assessment and rule management. What that changes in practice is where your stewards spend that time. They make the judgment calls instead of handwriting rules, and someone who is not a data quality specialist can still get useful work done. The third one covers two jobs, which is why it takes two columns. On the discovery side, it answered where the data came from in natural language against the catalog, so users find assets and understand the context around them. On the glossary side, it automates how business terms get defined and standardized. That is the unglamorous half. And it is the one that decides whether active customer means the same thing in sales and in finance. Or, to go back to Meridian, whether closed claim means the same thing in operations, finance, and leading. It was never a technology problem. It is a documentation problem that nobody ever had time for, and that is precisely the kind of problem an agent is good at. One claim to close on. And I say it now that you have seen them wrong. These are purpose-built agents for data engineering, not a general-purpose assistant pointed at a data platform. This brings us to the Qlik MCP server, which ties everything together. It is generally available, and it covers nine tool categories, more than 70 tools, all under one protocol. And it is not locked to a single AI tool. It works with Cloud, ChatGPT, Copilot, and any other MCP client. Let me walk through the categories, but grouped by what they are actually for, because nine bullet points read like a list, and this is not a list. First, finding things. App discovery and metadata lets you find apps, explore their structure, and inspect fields and sheets. Data lineage traces data origins and the full transformation chain. Between them, they answer, what do we have, and where did it come from? Second, trusting things. Datasets and data quality lets you inspect schemas, profiles, trust scores, and freshness. Business glossary lets you create and govern terms, categories, and data relationships. And master items gives you reusable, governed dimensions and measures, so that revenue means one thing across the organization rather than 11. Third, using things. Data exploration and analysis covers ad-hoc queries, field values, and chart data retrieval. Data products lets you create, activate, and distribute curated data packages. Selections and filtering lets you apply, inspect, and clear filters across apps. And visualization and sheets lets you create sheets and charts, filters and KPIs, which is exactly what you watched happen a few minutes ago. That's the full toolkit. Nine categories, one protocol, and it's available today. And the reason I keep coming back to one protocol is that it is your exit option. If in 18 months your organization standardizes on a different assistant, the integration does not get thrown away. That is worth more than any single feature on this slide. Thank you for your time and your interest today. If you take one thing away, let it be this. The bottleneck in most data teams was never intelligence. And it was never effort. It was the distance between somebody knowing what they need and somebody else having the time to build it. That distance is what these Asians close. Everything else on these slides is detail. And the honest measure of whether this works is not how impressive a demo looked. It is whether in six months anyone at your organization is still quietly keeping their own private copy of the numbers in a spreadsheet. When that stops, you have won. At Qlik, we believe in doing data differently, and I hope today gave you a clear sense of what that looks like in practice. If you'd like to see this against your own data rather than Meridian's, that is a conversation we are happy to have. Thank you.
