Transcript
My name is Max and I head education and community for Okta Workflows. Today we have a guest speaker, Jorge Belmar from Ploxi, and Jorge will talk about setting up Okta Workflows alerting with Datadog. Just a few housekeeping items. This event is being recorded, and then we'll send a message tomorrow with the link to YouTube where you can watch a replay of this event. If you have any questions, use the chat window. There is also a Q&A section, whichever you prefer. I think that's it. Jorge, I'm going to give you the stage. Perfect. Thank you, Max. Hello, everyone. Thank you, Max, for having me. One second, I will share my screen. Let me stop sharing here on my end. I think that's probably. Here we go. All right. Now I can see your screen. Okay, perfect. Thank you all for being here today. Thank you, Max, for having me to share this small information peel about Datadog and Okta Workflows. Today's talk is about proactive real-time monitoring for Okta Workflows using Datadog. You will realize later that you can do it basically with any tool, but I was just using this one for the purpose of the demo. For the ones that do not know me yet, my name is Jorge Belmar. I am based in Madrid, Spain. I work as a computer engineer, and I'm currently entitled as Global Identity and Access Manager for a French company. If you are interested, you can always connect with me in LinkedIn. Today's agenda, I will show you a small disclaimer about the presentation, mainly about the scope and the things that you need to trust or you can rely on it. I will show you the prerequisites that you need to have in place in order for you to implement this integration. I will show you what is the execution log streaming feature in Okta in case that you are not fully familiarized with it. I will show you how to configure it. Once you're able to send the logs that you are generating in Okta Workflows to Datadog, I will show you how to process them with pipelines so that way the information that you are transferring and generating there can be useful for you to create monitors and notification rules and really take profit of the productive monitoring and do the right things when specific events like errors or warnings are happening. I will do this very quickly. Later on, we will move directly to the demo time. By the end of the session, we will have a Q&A slide so you can freely raise any kind of that that you can have. With disclaimer, I am not a Datadog expert. I'm just an Okta customer that was using a specific observability solution like Datadog. All the things that you will see here are based on my own experience, so do not take this as written in a stone. If you are planning any implementation project, please always feel free to contact the official support or professional services, either from Okta or Datadog. Having said that, let's move to the projects. Of course, you will need a Datadog account, otherwise, this won't be happening. But as I said before, you can use any other observability solution. You can use Splunk or basically any kind of HTTP endpoint that can support this. Of course, you will also need an Okta Workflows account. Just be aware of the limits because depending on your commercials, you will have different restrictions on it. You also need super admin access. This is just for the initial configuration. For you to put the right settings to send the events to an external tool, you need to have this form. It can be assigned in a temporary way or you just need to find the right people to execute the configuration. This one, very quickly, you will need to get an API key in Datadog. This is quite easy. You have the outline in here for you to check it carefully. I will show you later how to do it. But you just need to go to your Datadog account, press your name, go to organization settings, API keys, and generate a new one. Just be really careful. Normally, people forget that API keys are quite sensitive. As an advice, use a dedicated API key for this purpose. Always use different API keys for different environments and different scopes. Because it's not the first time that I see people using the same API key for everything. If for some reason the keys leak, you will be running in circles trying to rotate it everywhere. That's it for the API key. What is execution log streaming? This is a relatively new feature that Okta released last year, that it will allow you to send the execution events of the Okta workflows to an external observability tool. As I said before, you can use either Datadog, Splang, CloudWatch that was recently released in March to receive information in an HTTP endpoint, or basically any kind of HTTP endpoint that you have prepared or built in-house to receive this kind of instrumentation. How to configure it? Before to configure it, please make sure that someone else has not configured this yet in your organization. Otherwise, probably you will be breaking something. In order for you to configure it, you just need to go to your workflows portal settings and go to the specific tab of execution log streaming. In there, you will need to enter just a few pieces of information. This URL for Datadog, I will show you later in the demo where to get it according to your location, because if your account is located in Europe or in US, it's a different URL. And then you have the request headers. Please here be aware that you will have two different sections for the headers. One is for the encrypted headers, and the other one is for the optional ones. So make sure to put the Datadog API key in the encrypted one, just to be sure that the key is safe. And then you will need to put a request body. This is just a JSON extractor that is wrapping the whole of the workflow metadata event that is happening specifically here where it says event. But we are wrapping all these in a different extractor. So that way, Datadog can fully understand where this information is coming from, what service is related to, what is the source, what is the host name related. So that way we can better fine-grain it or categorize it later on. For the processing and pipelines, we're using the syslog severity levels. Basically, we are just using a few of them, the debug, just for information that you are optionally generating in your own workflows. Let's see if we have time to show you this one. But the built-in information that we're generating using the execution of streaming will be info for the flows that you are starting or completing normally. Notice, for example, if you have something that it could be need to be checked carefully like a flow that was canceled. Probably it's not something so important to generate a warning or an error, but maybe someone will want to take care of, check why this happened. In addition to that, you have the most important ones that are warning an error. Normally a warning, for example, we can generate it if a flow is throttled because of the capacity. But I would say the most important one is the error when a flow is really failing. So basically the main purpose of having this product in monitoring is for you to know before of your customers that something is failing. So if you can identify something on the first error, probably you will be avoiding 50 more later on during the day. So something important that you can see in the bottom part is we will need to process and transform some of the information that we are sending to Datadog because by default Datadog is mapping the known attributes or values from the industry. But this flow type metadata attribute that we are sending from Okta is not a SOA standard one. And that's because we will need to manually map the information to the right directory in Datadog. Once we have the information fully created on the Datadog site, we will create the log monitor within a specific query where we will be filtering by service, by source, and for the specific status of the log that we are receiving. You can also set thresholds. This will depend on your needs. You can say, okay, if we are receiving during a cumulative time of one error or one warning, it's a warning state. And you can define a specific notification channel or rule according to that. But if you have, I don't know, 200 errors during one hour or less, probably you will want to notify a different team or you want to escalate or maybe use a different channel. So altogether with this, you will configure the channels in Datadog. You have a bunch of different integrations. For the purpose of the demo, we will be just using email, but you can configure with Slack, Microsoft Teams, PagerDuty, Gigantic, ServiceNow, and so on. So let's move to the demo. Let me share my other screen. Okay, so first thing that we need to do is go to the Datadog portal, check your name, go to organization settings, API keys, and in here, you will generate a new one. So let's call it Octa-workflows-execution-log-streaming, create key, I will copy this one to a safe place. Once you have it, please do not commit it to any repo. So let's move to pipelines. Let me come back to the Octa portal. So here I am in my Octa-workflows admin console. I'm going to settings, and as I said before, I'm checking that no one else has configured this before. So I will edit the configuration. I will double the enable-execution-log-streaming, and I will go to the Datadog official documentation. In here, you will see that you can select the Datadog site of your account. In my case, my account is located in Europe, so I'm selecting this one, and this endpoint will be changing according to that. So just need to copy this, put it in here, and then you will select the events that you would like to send from Octa-workflows to Datadog, because probably some of them are just useless for your purpose. Maybe you don't need to notify when a flow is starting and completing, and you're just interested in the tricky ones or the important ones like the flow fail. So you can decide whatever you want, whatever is needed for your specific use case. So this is the authorization header encrypted section that I mentioned before. In here is where you need to put the API key of Datadog. Let me copy and paste. This checkbox is designed for you to decide if the default behavior of the newly created workflows will be having this lock execution streaming enabled or not. So by default, I prefer not to have it because sometimes you are just trying out something new, validating a new development. Believe me, you do not want to worry the operation teams. So it's better to enable it just when you are sure that the workflow is working as expected. Additional headers, optional ones, but recommended by Datadog. So accept and run the type. You can find all this in this documentation. And finally, as you can see by default, Okta is just sending a wrapper of the whole metadata in this single event. But as I explained before, we will change this to a different structure that will allow us to fine-grain the information on the Datadog site. So basically, let me save it because it will be more visible. Okay, so we have Datadog tags. In here, I'm just specifying environment production and service Okta. So basically, in Datadog, you have a way to properly split the different services that you are observing. In this case, it would be Okta. And the source would be Okta workflows. In some organizations, you can have, for example, different sources for Okta Active Directory agents, the native integration for system blocks that Datadog also has. So you need to specify the source to correctly filter the specific events that we're going to generate. So here we are specifying, once again, the service because otherwise, if we just put it here, it won't represent in the tags. We will need it later. I'm directly specifying the hostname given that we're configuring this in the organization. There won't be another hostname and it's easier to configure it this way. And in this message attribute, that is our reserve word in Datadog, you can check it in here. So if you want to look for message, you will see all the different fields. And you will see that this is our reserve attribute where Datadog will be receiving the JSON extractor with the events. So now that we already save the information, I will send a test event. So I will send a flow start. You can see a green message in the bottom part. So it's fine. Let me now send a flow fail. And also a flow trouble. Okay. So I sent three different events. I'm going to Datadog. You need to go to logs, explore it. And you will see, let me put it in live mode. So I already received two of them. As you can see, flow start, flow fail, and flow trouble. The problem that we have at this point this one is info, this one is info, and sorry, and this one as well. So there's no way, even if this flow is failing, for me to recognize in Datadog that this is the situation. So that's because we need to put in place these pipeline processors to correctly transform the information. So basically a pipeline applied to the logs is just a combination of actions that will be performed to the log between being received in Datadog and being shown to you in the console. So we already have here the Okta Workflows, the built-in pipeline that comes with the integration, the native integration, but we're going to create a new one. We're going to call it Okta Workflows ELS. And we are going to use a query. So as we send this information already with the right tags, we'll be quite easy. We'll be generating specific query for our purpose. So all the logs that comes from Okta Workflows that belong to the Okta service, it will be processed by this pipeline. So I will create it. And from here, I can add a processor. So what I can say is, okay, if you see the metadata that Okta will be sending, you can see the base schema of the event that we'll be sending. So in this case, you have this event type. And event type is not a native attribute name for Datadog to determine that this is the status of the event and that's because we need to map it. So what we will do is we will select Lookup. And we have this snippet that I already prepared. So basically what we will be doing is, if you are receiving an event type with flow fail, we will consider it an error. For flow throttle, it will be a warning. And if someone is canceling a flow, it will be just a notice. We can configure this fallback. So everything else like a flow start, flow complete will be considered as info. So now we need to determine in the incoming event, what is the name of the attribute? So as you can probably recall, the name of the attribute is event type. And we will set the target attribute as status. So with this, Datadog will know that in the status attribute, it can correctly map the new status. So we will add a new processor. We will map the status of the event to the status attribute. And that's it. So now if we come back to the Deluxe Plural, we put it in live and let's perform the same test once again. So flow start. Flow fail. Flow throttle. And that's it. So let's give it a second. There you go. So we already have the flow start with the status info. That's fine. But now if we receive the flow fail, it will be an error. It will be an error. And if we receive flow throttle, it will be a warning. So now we have the right categorization in Datadog for us to work with it. Let me do something very quickly in here. I will go to flows. Let me create a new one. I will call it first level. This is the thing that I mentioned before. You need to manually enable this option for you to send the information. So now what I will do is, I will do something that is not allowed. I'm going to divide by zero. I will execute it. Let's give it one second. So what is expected now is because all these were generated with the jazz test that you can generate for the integration. But now we will be receiving our first real event from Octa2Data. So as you can see, it's already considered an error, but we have the whole metadata of the event here below. But now you can also see that we have new attributes like the error message and the error type. You also have the execution ID, which is the one that you can normally see here to be back with the execution history. So as this information can be useful, we're also going to map in the data that you can see in the documentation that you can map the error message and the error kind. So let's move on to, once again, to our pipelines and add something else. So let's say remapper, and we are going to remap error kind. We're going to map it to error kind. I think the default error type, my bad. And for the error message, the same. So we're going to say error message, error.message. So, and for the trace ID, it's quite interesting because if you are generating a lot of execution or child execution from one workflow to the other, it's great to have a single ID that is grouping all those execution in a single trace. So that way, if you want to filter all the flows or the execution associated with a single execution, you can do it. So in here, we will put this one, the root execution ID plus execution ID, because this is giving you the execution IDs related to the parent one, in case that this is being triggered from a child event, or the execution ID directly if this is a single flow or a parallel one. So now if we go to the Explorer and we do the same, let me re-execute this failure flow. And you put it in live. Okay, great. So we have the info of the flow start, but we also have the error. And now you have new things in the UI. First, you can see the error kind. So it says that it's invalid input. So you can better filter the logs because of this. And you also have the trace ID. In addition to that, you have the error message clearly stated in here in red. So you do not need to check the whole metadata to find it. If you press in the trace ID, you can also filter. So if you filter by this, you will see all the executions associated with the same execution. And in here, considering that the same host we can remove this column, we can remove the service and we can say, okay, for example, I want to show the trace ID in the columns. And also I would like to know what is the flow that was executing this. So if I want to have it here, I can later on build a dashboard or a better view to have this direct observability in a monitor or whatever. Okay, so this one is properly showing me the errors. Now, let me create a second one, second level. This one will be a helper flow. So we'll be calling from the first level one to the child flow. Get this one from here. Let's go to the second level once again, and we can probably put the errors here. So let me execute it once again. Let me remove this filter. Okay, so here we already have the differentiation between the first level and the second level. You already know that there is an error in this level, but the good news is that you have the whole trace ID to map it. And now we can go to the monitoring that is the product monitoring part, and you can create a new one based on the information that we already have. So in data that we have a lot of sources where you can get the information for the alerts and the monitors, but in this case, we will be using logs. So in here, what you need to do is similar to the previous one, you just need to write a query. And in this query, you will be specifying that you want the service Okta from coming from Okta workflows. And you can say, okay, this particular monitor will be for the environment protection. You can split them. And in here, we will evaluate, I don't know, five minutes. You can do different rolling windows or cumulative windows according to your needs. Now in this case, we'll use last five minutes. I will use for the thresholds, just for the purpose of the demo, I will put two errors will be an alert over or equal one will be a word, just a warning. If we don't have data, because if you do not have constant log execution, it will be just no data. Let's evaluate it to zero and will be considered as okay. So for this one, we can call it Okta workflows ELS production and we can put a key. This notification, and that's it. So now we can just, oh, I forgot something. We need to put the tag. So it's the same, just for the notification rules to know what monitors they belong to, or they should be routing to, we need to specify once again this. So let's say that this is service Okta source of the workflows ELS. And let's also specify that this is related to production. Something else that you need to specify in the query, otherwise it's going to fail, is what are the status that you want to consider? So in this case, we will be considering just the status related to error or warning. Any other won't be measured by the monitor. And of course will be considered as okay. So we just need to save this one, create and publish. And we'll start reading. You can put the transitions in here and it will show you that was created. It's already in alert because the errors that we raised in the workflows. And now in the monitors, you can come here, click on settings, go to notification rules. Add a new rule. And once again, we will filter by Okta source, Okta workflows. But in here is different because if I have, for example, the same team working in both environments, there's no need for me to specify additional tags. So if it's matching at least those, this notification rule will be valid for the monitor that I already created. So in this case, on the right part, you can see it says one matching monitor. So for this monitor, this notification rule will be taking place. So I do not want to notify always because when the situation is okay, there's no sense to receive an email. But if there is a warning or if there is an alert, I would like to receive an email. As I said before, you can configure additional channels. Right now, we're just going to use the email. And I can say Okta workflows too. So I can just create this one. Let me adjust the duration of this to make it more fluent. I will say custom rolling window of maybe two minutes. So let's give it some time for it to refresh. To come back to the okay state. Meanwhile, let me show you something else very quickly. So this is for the built-in integration between Datadog and Okta workflows. So this is using just the metadata that is defined, predefined by Okta. But if for some reason, you also would like to send business information or custom debugging information from your flow, you can also do it using a dedicated helper and an API connection that we'll be doing basically the same that we did before. But let's go back to Okta workflows, connections. Let's create an API connector. Let's call it datadog.ingestion, custom authorization. Getting the API key. So it's created. I'm coming back to my helper datadog. So it will be creating first our headers. We see that we have accept our content type. And for this one, we'll be building an object that will contain dd-tags, service, message. If you recall, this is the same information that we put in the native configuration of Okta workflows. So in this case, we'll be sending the same information. You can, of course, add these in some sort of parameters table or something like this. I will create merge. Merge. We'll be merging these. Let's put this one. Headers. So I have these, and now I need to merge these with another attribute that I will put in here. So we just need to put the status. So we'll be mapping this with this. The flow name. So we can make it consistent and have all the information that we already have for the built-in integration. So this is the color. And the name. So right now, we will be merging this. And we will put the message in here. Sorry, this is not the same message. So now that we have it, we will execute an API call. We will send to the URL that we checked before in the documentation, same that we used for the initial configuration. We will put the headers that we created over here. And the body that we just created. So with this, let's go to the first level. Let's go to the second level and remove the error. So it's fine now. And in here, let's call flow to our helper. So in here, for example, let's send this debug status. And let's build an object related to our own flow. For example, we can say we're creating a user. First name. Last name. And we can say John Boo. Okay, so we have this object. I mean, this object in here. We'll execute it. Let's see. Okay, so if I go to the execution of the helper, you will see that receive the debug status, the information related to the flow, it merged in a single object, and we have it in here. So basically this is the structure that we are sending to Datadog. So now if we go to Datadog once again, let's go to logs, explore it. You will see that we have this debug information in here, and it's containing the information that you sent. So basically you are not just forced to be using just the information that OctaWordFlow is sending, the transactional ones predefined for the different states of the flows, but you can also send a specific information that can help you to better debug. Remember that the information of the execution history just lasts for a while. So if you need to check these months later, probably this information can be useful for you. So make sure to respect the privacy and the sensitive information, but it can be useful. And given that you can also filter in here, I can say, okay, I want to check all the logs coming with first name John, and it will show it to me. So if I come back to the monitoring list, I can see that this one is once again in okayed state. Normally I prefer to check it in transition because it's more visual for me. I can put past five minutes. This has been okay for the last five minutes. I will execute once again, the first level, but this time with an error. Based on my experience in some cases, Datadog takes a few seconds, in some cases, even a few minutes to really process the monitor from the okayed state to the warning state, because it needs to go from Okta to Datadog. In Datadog, it needs to go to the preprocessors, the pipeline processors that we build. Later on, the monitor needs to check the query. So in some cases, take a few seconds. So let me refresh. But still, maybe I already have some before. Maybe not. I will generate another one, just in case, so we can reach the alert state. This one was expired. The one. Okay. So at some point, this should be transitioning to warning state. As I said before, using the notification rules, you can define if you will route the different states, like the warning or the alert state, to different teams, different channels. That is totally up to you. You also need to define what is the threshold that you would like to define. This one is already in one state. So I should be receiving an email. There you go. So we already have it. Even though I just put this notification, I won't be getting deep in how to configure or customize the email, but there's a specific article in Datadog fully explaining what are the handlebars or the placeholders and the variables that you can use to put more information about the specific incident. But given that normally you are consolidating multiple errors in a single email, it's probably not so useful, and it's better to put this kind of built-in link for the engineer to be checking directly in the Datadog console. So that's it. Let me come back to the presentation. One second. Okay, so I'm not sure if you have any questions about this one or something that you would like to better clarify that was probably not clear enough for you. Hey Jorge, this was wonderful. I know Workflows had this capability for some time, but we never did. So I'm not sure if you have any questions about this one or something that you would like to better clarify that was probably not clear enough for you. So I know Workflows had this capability for some time, but we never did a meetup about it. So this is really good. And then as I said, we'll put this on YouTube and then so you can return to it and watch it later if you would like. But yeah, let us know if you have any questions also. Jorge's information is on the screen. If you have any additional questions or doubt about the things that I just present. And as I say, you can connect these to basically anything at the moment. So feel free to explore it and take profit of it because the proactive monitoring is basic if you have multiple workflows. Well, I don't see any questions. So yeah, and I guess people can always contact you on LinkedIn or via email. But if there are no questions, Jorge, thank you so much for hosting the event with us. Thank you for having me, Max. Thank you everyone for coming. This was wonderful. And yes, thanks everyone for coming. Like I said, you'll get a message tomorrow with the link. And yeah, we'll see you at our future event. Thanks again. Sounds good. Thanks everyone. Bye.
