Transcript
Hey, guys. It's Mike with Qlik. Thanks for joining me in the next video of the Do More with Qlik video series. I'm excited to be here in part five of the series on data products for Qlik Analytics, where we've been exploring data sets as a core component of a data product. In this section, we're going to explore semantic types and data validation rules, which help identify the issues in the data, highlighting problem areas so data stewards know exactly where to focus their efforts. So let's jump right in. So we'll pick up where we left off within the data catalog, looking at our data sets. Now, for this demonstration, we're going to use a new one named Line Items and Fulfillment, which basically combines order details and line items table. And I just used Dataflow to do that. So let's go into Line Items and Fulfillment. And just like the previous videos, we have our user interface with our overview, profile, data preview, lineage, impact analysis, and validation rules. Now, a quick note on data sampling size for profiling versus data preview. When you apply data validation rules or semantic types with validation, you're going to want to sample the values within your data to see which of those values are valid, for example, against those rules or semantic types. Here we have percentage of the data set 100% or number of rows. This is different when compared to looking at data preview, which is just looking at a preview size of either 100 or 1,000 rows. I have mine set to 1,000. If you're curious to where that is, it's in the ClickCloud administration under Settings, Feature Control, Catalog Dataset Preview Size. You can set it to 100 or 1,000. So keep that in mind. So I mentioned that we're going to focus a lot on the Data Preview tab as well as validation rules. When you look at validation rules, you're going to see that there aren't any applied. And I can click Apply Validation Rule. And if there aren't any that exist, you can create it manually. Or you can actually have AI create it through suggestions, looking at the actual columns and descriptive information to make some recommendations. And we'll talk about that later. The reason I'm mentioning this is because if you go into the menu, you're going to see a section named Data Quality. I'm just going to open that up in a new tab. And Data Quality basically is where you manage and organize the different data quality objects, such as semantic types or validation rules. It also allows you to look at weights and define those weights for the click trust score for the various dimensions. Validity, completeness, you can even turn on or turn off other dimensions, et cetera. This is going to vary based off your organization. But if you needed to configure these weights for the click trust score, you can do that here as well. So keep in mind when you make changes to your validation rules or semantic types, you're going to do it within the Data Quality section in the menu. And we'll come back to this once we set up a couple of them. OK, so coming back to the data set, let's look at Data Preview. So the first thing we're going to do is we're going to focus on Vendor. And we're just going to set up a very simple semantic type for Vendor, and I'll explain why. So the differences between setting up a semantic type versus a validation rule, semantic types basically ensure each column contains correctly formatted data. Think of an email or a phone number, while a validation rule will ensure the data is logically correct based on business requirements, including checks across different fields. So the first thing we're going to do is just set up a semantic type for Vendor so you can see how it works. OK, so let's work through this. I'll click Vendor, and you can see that we have a description and the data type, which was set up as a string within the profile, or it's defined here as text, doesn't have any typical semantic type set up with it, and it doesn't have any validation rules. So going to Data Quality, under Semantic Types, I'm going to create a new semantic type. And we'll give this a name, and we're going to call this Vendor Check. Can give it a description, and we can choose if we want to click Trust Score to use this for validation. And then there are three different types. There's Dictionary, which validates values against a predefined list of allowed entries. There's Pattern, which validates values based on specific format or structure using a regular expression. And then there's Compound, which basically just combines multiple semantic types together. And to give you a quick example of that, if I search for URL, you can see we have a couple of semantic types, Mail, Data, Website, HDFS, and File. And these all are defined as patterns. Now we have one called Generic, and it's defined as Compound. So if we look at Mail, you can see there's a regular expression for the pattern for Mail. If we look at Data, you can see there's a regular expression for the pattern for Data. If we go to Generic, it's actually combining Mail and Data, as well as File, HDFS, and URL. So if I click Edit, and I wanted to add a new one, I could just select from an existing semantic type. So that means if the data column it's checking is set up as a semantic type of Generic URL, if it contains any of those pattern formats in that column, it'll match that particular rule or semantic type validation. OK, so let's go back to the semantic type we were setting up, and we're going to set it up as a dictionary. So what I've done before we select the values, I wanted to show you that I have a little app here. And this app is just looking at the unique values in the Vendor column. We have Commodore, MSX, Odyssey 2, Retro Game Boys, and Retro Game Boys LLC. So out of these five values, Retro Game Boys and Retro Game Boys LLC, these are vendors, and these are brands. So there's obviously some issue in that data that will have to be cleaned up. So what we could do is create a semantic type defined as a dictionary with a list of allowed values. And those values can be Retro Game Boys and Retro Game Boys LLC. And there we go. So that sets up that semantic type. So I click Create. We're going to have it used for validation for the click trust score. And what we can do is just go back to the catalog. And all I'm going to do here is click Refresh. And when I do, I want you to keep an eye on the Vendor column and keep an eye on the data type area. So the refresh is now running that resampling of the profile. And it will automatically look against the semantic types and data validation rules and make those adjustments. And there we go. So now our data type is set as Vendor Check. And you can see through the visual graph, now we have 89.9% valid values, 3.5% empty values, and 6.6% invalid values. Remember we talked about the sample size? Let's go in here and let's change number of rows just to 100 and click Refresh. So now it's using only 100 rows to compute the validation. And you're going to see a difference in the visual indicator for Vendor. So notice that our null values are now missing. We don't have any black in there. There's 93% valid values and 7% invalid values. And if you click on this, it'll actually filter on those as well. So here we have Commodore, MSX, Odyssey 2, et cetera. And then here are our valid values. So a nice little demonstration of changing the sampling size and seeing how data validation works as a semantic type. OK, so we're going to wrap up this video now. And the next video will cover data validation rules. I'll see you on the next one. Bye.
