Transcript
Jay and I have actually had an opportunity to spend time together for a couple of years now. I guess we met at that little event with, what was it, the Dallas Cowboys thing. We got to sit down and have lunch and talk about where you're going. I think at that point you were at, what would you say, a zero on the journey? Ground zero, that's right. Ground zero. And so in two years, so this is kind of a chance to talk about what two years of the journey looks like. And if you're on your own kind of automation journey, I think this is going to be very instructive. So before we jump in though, let's start with just who are you and who is Oxy and we'll just kind of set some ground rules, I guess, and then we'll get started from there. Okay, so for those of you who don't know, Oxy is an energy company. We've been around for about 100 years. Actually last year we were celebrating 100 years. And for those who are baseball fans, Astros, you'll see our logos on the left side or right side of the pitchers or the batters. And you see our logo there. What we specialize in is around energy. So there's three domains. We have energy segment, we have the chemical segment, and then our latest one, which is a low carbon venture. I'm going to slip in a little gig here, it's also our direct air capture is actually opening next year. So that's going to be our big ventures into what Oxy is. That's awesome. That's awesome. And where are you starting from on that journey kind of two years ago at Dallas, in Dallas? Like where are we at? So just to give you a little bit background, Oxy has always been an on-prem company. So everything done is we have a very specific teams, groups working in very silo. He works on security, firewall, working on firewalls, and compute and storage same way. Yeah, absolutely. Very silo. And a lot of customers are facing kind of a similar thing, and as you look at your process and how you release software, I think we often assume that it's probably better than it actually is. When we actually look at it, we realize it's worse than we thought, but you have a feeling about how things run. And a lot of customers find themselves in that spot, and it can seem overwhelming. It feels like we have to modernize everything, load, bounce, or fire. There's a thousand moving parts. How did you start to look at this and refine your thinking and begin to prioritize what you needed to work on first, second, third, fourth? That's a loaded question. So I'll start with those of you who actually saw Simon Sinek, the golden circle. So the way he started was started with why, how, and what. And then my suggestions to you guys all is to really identify what those why is in the first place. For ours, it's more about time to value or mean time to repair, which is MTTR. Those are our why. And what we're aiming to do on that specific thing is to enable the business time to market, and that's what we're here for. And that would be addressing the why. The how is, I think this is where you hear me repeating over and over again, is around the business impact assessment. We went through and identified these things where we didn't realize how long it takes for us, our on-prem process, to implement. I'll take it easy, too. It takes quite a time, and I'll show you in the slide a little bit how long it takes for our process. Once we identify the how, then we go to what do we do and how we address these things. So maybe that's a good place for us to begin the setup there. What did it look like at the beginning from a process perspective? And then given your metrics, TTV and MTTR, how did you begin to break down the process? How many stages did you have? Where were you trying to take it? How did you determine which stages to address first? From that perspective, we're trying to take it more of a phase approach. So first we identify what the... We basically break down to different phases, the immediate actions, the short-term goals, or the long-term goals. So we know that from a short-term goal, we need to have buy-in from upper management. Because I could imagine you all probably have the same problem where we have this big boiling in the oceans problem to solve. So my suggestion is focus on immediate buy-in, socialize with your peers, how to address these things. And then the second item is the short-term goals. So from a short-term goals perspective, I would look at where you are in the stage, identify where the gaps are. And then addressing like, for example, for our process, we built what we call a platform engineering teams. We built that. We also have the... Start building what we call identify as a cloud operating models, which help identify where the team who is accountable for what. And then the last one we would look at is the blueprints. So I know that you guys heard from Aman's discussion this morning keynote. He emphasized on having the golden pattern or what internally we call as blueprints. As a long-term, because we are essentially a long-term on-prem company. So what we're trying to focus is how to up-skills our workforce to be able to support these long-term. Yeah. And maybe if you want to pull up the graphic and kind of talk through kind of that early stage, describe for us what this looks like as your before and after desired state. Certainly. So as a problem we have today as a cloud, I'm sure, again, like I mentioned earlier, we're startup as an on-prem company. So we encounter multiple things. We have a lack of cloud knowledge in our organizations. Also the bare entry into utilizing the infrastructure as code is also somewhat very difficult for a non-developer. If you think about it, you have to learn how to use Gits, you have to learn how to use Azure DevOps, you have to learn how to use Terraforms, and not to mention how to do pipelines. So you have all these things that are going against you. And as a start, we were trying to do the problem that we had is people don't know what they don't know, so they start click-ups. And that's the problem we always have. And I'm sure, you know, the reason why you guys are here is how do we get rid of click-ups and try to automate this process? That's right. And I see in the legacy estate, you had 222 days. How many phases was that to consume 222 days? So I'll show you the next slide, item one here, which is this is a... We actually asked HashiCorp to come in and try to do an analysis on our process today on our on-prem. And this is more of an eye-opening is this is to request a EC2. As you can see, it breaks down to all these tickets that we have to submit. And the colors on the bottom is how many different teams it involves in each of these processes. So as you can see, it takes 74 days for us to provision the servers today, assuming that you know which tickets to open. And this is only for one environment. So you can imagine what type of shock we were once we found this out. So this is a little bit of what Susan had mentioned about kind of a little bit of a white-glove assessment, where we just wanted to ask the various stakeholders, you know, what are the various stages? What's the value that's added at each stage? How many teams are involved in each stage? And we essentially kind of just draw it out and say, here's what we think it looks like. Do you concur? And then we kind of go back and forth on what it looks like. And ultimately, when you get your eyes on the actual pain point itself, it really jumps off the page and makes it a little bit easier. You and I had talked about this before, but oftentimes I use the... Anybody read The Goal or heard about like the theory of constraints, how to look at a bottlenecking system? Yeah. So a lot of times what we end up kind of attacking this with is we look at this type of a process and we say, where's the bottleneck? Where's the long pole in the tent? If you're familiar with the theory of constraints, the idea is that you can go as fast as you want, but if the bottleneck is... Let's say we can get an EC2 instance in 35 seconds, but if it takes me six weeks to get traffic to it, but, you know, like, great, I've got an EC2 instance running someplace, but I can't touch it. So all of the work piles up against that networking team until you're back at the same TTV, right? I can't make use of the thing until I get the bottleneck out of the way. So the idea is like prioritization starts to look like, where's the long pole in the tent? Maybe we automate that first, drive that out of the pathway, and now we get a better TTV. Is that somewhat what you applied or some of that thinking entered into what you were doing on these various stages? Certainly. As part of the... I know, Michael, you were asking earlier about how to get the buy-in or explain the why. I would encourage you, if you can work with HashiCorp on doing the BIA, right, the business impact analysis, I would truly encourage, because this is... You don't need to do the why selling. This itself would do the selling for you. Yeah, and how was it working with the team, by the way? It just says... Oh, it's excellent. I would, again, can't say enough. I would emphasize that. And I think to your earlier questions is, how do we solve each of these steps on the way to make it identify where the low-hanging fruit... Because as you start building this and you realize that a lot of the work that we have been doing and the team are actually sitting on the second row here, so if you have some technical questions, I would encourage you to talk to them. I'm more about change process. We're addressing the process and people and things like that. We try to identify things that are low-hanging fruits so that we can have a buy-in from upper management and show a quick win. So that's to your point is, where do we identify these things? It would be something that's a low-hanging fruit that we can have a quick win. So that way, we can then show off to the next group, say, okay, now, if we can automate the next phase, now we're lowering our time to market even lower. Yeah, I love that. And the idea of... I think Sarah was just talking about metrics and marked improvements, the ability to catch something that's low-hanging fruit that you could automate quickly and easily, I always think in terms of that quadrant exercise where it's like impact to the business as the vertical axis and then cost or risk or difficulty of implementations across the bottom. And anything that ends up in the upper right is something that we can prioritize as low-hanging fruit that delivers significant value, show it to the leadership, and kind of get additional buy-in. And this brings me back to one of the things that you mentioned earlier was the need to drive top-down sponsorship. It sounds like this was a good way to get that top-down sponsorship. And then from there, you're able to accelerate a number of things. Exactly. So we're trying both, actually. We try from top-down and from peers. So that's our goal is to enable the team. But you're right to your point. Being top-down helps tremendous. Yeah, for sure. I'm assuming you never hit anything in this process that had political implications. It was just all easygoing. I wish. And again, this is recorded, so I've got to be very careful what I say. No, we do run some challenges a lot. But again, I think I'll stick back to my suggestions is do a quick win, provide a process showing the efficiency and how quickly we can. The area where I'm not sure a lot of you guys encounter this, but our leadership trying to retain workforce. So the challenge has always been, how do we upskill individuals to update to the cloud? And that's always been almost the challenging things that I'm sure everybody here is encountering is how do we make sure that the securities is having the proper guardrails in place? How do you make them feel comfortable with the implementation where things are no longer out of their control? So one thing that I think is an eye-opening process for us is the fact that when you go to the cloud, it is no longer a silo tasks. It is a cross-functional process. Once you have this understanding, I think you'll have a better understanding and support how do we move into the cloud quicker? Because I can always blame, oh, it's a security, oh, it's a network. But if we work together as a team, I think that's where we accelerate and you have to buy in. Yeah, and that seems like one of the benefits of breaking down these various stages is the liquidation of the silos. We're handing work off. It becomes a more empathetic environment. Rather than them being the team that prevents you from moving forward, they become the team that's helping you move forward. Yeah, there's a lot of cultural benefits to that as well. Exactly. And one thing that really I think I want to add to it is the fact that by implementing what we call cloud operating model, I'm sure you guys are moving to the cloud, and one thing that we implement now is cloud operating model where we identify and say, okay, from each team, from each services perspective, we know this is what you owe, you're accountable for, you're support for, and these are the run books that you need to do. Once we identify those, the process or the barrier that you mentioned earlier has come down tremendously because now they know exactly what they owe, they know exactly what they need to support, and how they built all this process. Yeah, excellent. Now why, so as you looked at this, why was obvious, I guess I'll ask the question of why cloud? I still think that's a relevant question to ask. Why move to the cloud and why take advantage of that? And the second thing is why HashiCorp as it relates to that cloud journey? So from the cloud perspective, don't want to relive that again, but in 2018, 19, you're all aware of the pandemic. So we had challenges around the supply chain, and I became a supply chain expert because I need to figure out when the computers comes in, where I can buy all these stuff, and we have nine months delay. So since then, that's where we decide to move into the cloud. In addition, there is, I guess we can't have a conference without talking about AI, right? Otherwise, we're in the wrong years. So again, the emergence of the AI is when what we push us to the cloud, because we want to have ability to try things quickly and fail quickly, or try things and work quickly. So that's why we go to the cloud. Yeah, experiment with AI, get some high cost CPUs without having to buy them and maintain them yourself. Yeah. That makes a lot of sense. And then I think, to your point, right, we don't want to have 74 days provisioning of servers and try something in the cloud. So that's why automation becomes a critical part of our process that we're trying to do. Yeah. Interestingly, I feel like my wife became a supply chain expert during the pandemic. I heard about that a lot. And also the toilet paper. We know exactly when to come in and come out. Yeah. We had to know when that boat lands, when it gets unloaded. All right, well, so we know where you started. We know how complex it was and how you began to prioritize the work and work your way through it. Where are you now in the journey? Like how far along are you in the journey? And what have you seen in terms of material benefits from the work you've put in so far? OK. So no longer suspense. I'm going to show you what our next process would be like now. So this is our future states, what we're trying to do. So we're boiled down from 74 days to 30 plus days. This is our phase one, where we would build automations in place around using the Terraform cloud. We're using Packer to build images, and we're using Volt to implement the securities. And that really opened the door to security team, by the way, just as a note on the cameras. Once we had that Volt and just-in-time password rotations, that's when we gained a lot of trust from the security team. And as you can see, we're trying to take approach of the low-hanging fruits, right? Identify area where we have a quick control, quick win, implement it. And then our phase two approach, I'll go and show a little bit more, is we're trying to boil down from a 70 plus days down to four days. So that's our North Star goal. Now, I like that. Having an intermediate goal makes a lot of sense. A lot of people will look at kind of the Nirvana state and maybe not do a good job of staging goals or checkpoints along the way, and you always feel like you're still miles and miles away from the end state. Having something that you can near-term achieve and show and measure and talk to executives about saying, okay, we're down from 74 to, I don't know, 45. Now we're down from 45 to 32. Now we're in our first range, declare victory, and have a party, recognize the team, everybody, and then move on to the next stage and plan out the next phase. Excellent. So can you give us an example of, I'll get that right, an example of what constituted low-hanging fruit for you? So part of the example that we gave would be the entry point to AWS. Previously we have to do in creating an account in AWS. So what we do is we have to have a permission boundary. We have to provide a role attached to it. So our quickest way, quickest win was, hey, if you guys want to go and try something else, let's go and automate the AWS account creations, where we would give the teams, build it out, and you go to town with it, have fun. Awesome. Yeah, and they get to play around with something. That always makes developers joyful is when they can get in and get on a command line or preferably not click house. They can get in there and play around with something. That's awesome. And so as you communicated value, how do you communicate value? How often do you engage with executive stakeholders and those types of things? How do you drive that communication? So we keep constant, we have monthly meetings at Oxy where we keep constant update to the leaderships and also the rest of the organization where we are. And again, we celebrate. And what I really mean by celebrate, we provide breakfast, we provide goodies and things like that to make sure that they're on board with the entire process. Yeah. And now what about communication to developer teams and other operations? Is there like demo Tuesdays? How's that outreach look to drive consumption of the automation that you're building out? What we're trying to do now is the next item would be, to your point, I think it's trying to build what I'm on and during the keynote mentioned earlier is around building the what I call blueprints or the golden patterns, right? Instead of you engaged customers and getting feedback. So by building a blueprints, instead of you're focusing on reacting on all the new items that are coming in. Now, if you focus on building the right solutions to each of the application teams, then you're focused on that pattern. Once the pattern is being solidified, working directly with network team, working directly with the security teams, ensure that everything are done right, then you can release to the general public. Yep. And the actual other individuals, more like security network, they feel like they're part of the decision-making. So then they're actually buy-in. Also, the application teams are appreciative because they can do things repeatable again and they can do anytime without actually going through and filling all these tickets. Yeah. Very good. Well, as you look back, is there something that you would, maybe you can offer some advice to people who are early in their days on this journey? Is there something that you've learned through this process that you wish you could have told yourself two years ago? Oh, okay. So I think as individuals who are starting today and where you are at where I was before two years ago, my recommended number one would be to put in a cloud operating model as soon as you can. I think that would be my number one recommendation, is identify the services that you're going to use and work with the different teams. Identify what you're going to be the ownership of, what are you going to be accountable for, and what are the teams required to get involved? Because this is where you will get the biggest benefit. Because if you imagine, the traditional on-prem world is things are done in separate groups. So if you're doing a cloud, because one service goes across multiple teams, if you identify those properties, you can go through and identify, okay, security, you handle all the IM team security handles all the access permissions, networking, all this other stuff, network, you take over and you control, build policy, build sentinels around these things that you want to do. Now you have to buy in. They're not going to sit there and wait, or they're not going to now, okay, what am I here for the cloud? Once you identify that, that will be my number one goal, or in your shoes, put that in place ASAP. Yeah, it's really interesting. We've had discussions of this about kind of the shared responsibility model that is imposed in the cloud, right? Anybody heard the adage that if everybody owns something, nobody does? So if I say who owns security in the room, you're supposed to say everybody, right? But that seems like it's a little bit of a cop-out, because technically that means nobody does. We're all dealing with one or two things here and there. It's super easy to have a gap if I say we all own all the security constructs. So establishing the cloud operating model is a good way to really assess the shared responsibility model, who's responsible for WAF policy, who's responsible, like you get very clean on who's responsible for what pieces of the automation stack and how it rolls out. So I agree with you. I think it's really critical to understand ownership. Ownership is so key to driving this type of a transformation. So I'm glad to hear that you were able to kind of drive that way. Any other advice for the group? Sure. I have a few more. One of them is building the blueprints. Again, you're going to hear it repeating, is because how do you know which services to use in AWS? There's 300-plus services. You're not going to avoid the oceans to identifying those. So if you work with highly impact groups where we can bring this in and get a buy-in from everybody, showing the advantages of automation, that would be your quick win. So work on your blueprints, so that way you don't have to focus your time on reacting. Now you can proactively identify how to make sure the services are coming the way it needs to be. Again, easy to use. I think one thing that I kind of share with my team nowadays is that IT as a general is a service group. So sometimes we tend to forget that we are a service group, that we think we enforce too much power into it. We are a customer group, so we want to make sure things, when we build a platform, it has to be easy to use. You can build the best tool in the world, the most fastest API, but if you don't have the easy use, it's not going to go anywhere. So that would be my next item, is around building the blueprints and ensure that these are repeatable patterns. So that would be your next big win. And the cultural thing that you're talking about is really key there, too, because you need to see yourself as serving your customer, which is an internal user, and your job is to drive customer joy. Adoption. You want their engagement with you to be pleasant. Yes, exactly. And it's very important to do kind of customer interviews with that, or net promoter scores, some kind of survey, some type of contact. But how did you gather that type of feedback? So it's constant feedback. That's what we're trying to push for. So as we built the platform, the platform engineering team, where I'm really proud of my team here is we're encouraging different teams to try to leverage our process and then get feedback. Where are some areas that's difficulty? What are some areas where you'd like to see the features to help improve the process? So having that consistent conversations and communication help ease a lot of the process. Awesome. Any last thoughts? A lot. Yeah. For sure. One of the things is the cue cards are to help keep me under control. Otherwise, I'll turn this into a Joe Rogan three and a half hour dialogue. So this helps me a lot. But yeah, if there's any last thing, we got a few more minutes. Sure. Another area I think a lot of you guys are probably going to encounter is around the workforce development. And that's, again... Yeah. Skills. Exactly. Yeah. As I mentioned earlier, there is always a high entry barrier to our process, right? So you have to learn how to use Git. That's to me, from a non-developer perspective, it's a really difficult concept. Yep. The next item is how to use Azure DevOps. If people doesn't come in and they look at the applications, you're lost. It's like, what is pipelines? What is Git? What is backlog? Again, to you guys, it might be natural evolutions. But from a non-developer perspective, that's really hard. Pipelines. That is the whole automation process. It's always difficult as well because you have to learn how to use YAML language, how to do it. Or you can... Again, we're not going to ask them to do click-ups, right? The whole point is that we're going to try to use YAML and also Terraform. So learning how to use Terraform itself is also difficult. Yep. Yeah, a lot of the ideation that we have around the foundation of a platform team is often we'll take a couple of operators and we'll pair them with a developer because all of these concepts are fairly familiar to developers or people who manage codebases. But from an operations perspective, it's a little bit of a foreign language. So I have to build a little bit of a balanced team, which includes that developer persona that's more acquainted with source control and pipelines and those types of things. That becomes sort of your core unit for your platform team. And from there, you can build out as you add additional people to that, they will share those skills, the operational skills, impact the developer, the developer skills impact the operators, and you can begin to grow the team out that way. So depending on where they're coming from, from a skill set, you do want those disparate skills. Believe it or not, developers are not great with WAF config and network topology. There's areas of less information than others. So the idea of building out that centralized platform team, be thoughtful in that build out because it will help you develop these types of skills and will help you impact the rest of the organization with those skills. And then I think one last one I'll leave you guys is around having a CCOE. If you can organize and build a CCOE, that will help deliver the product where it is being supported organizational-wide. That's one thing I would highly recommend. Make sure you have a FinOps team as part of your CCOE. Make sure you have a procurement because sometimes a procurement can be a hinder. And have the technical people on it because that will really... If you start noticing that all these things are tied together, right? So how to build blueprints? Well, you go to CCOE, get approval that these processes are meeting our standards in whatever organization you're at, and then once that's approved, you start building blueprints. Golden pattern. Now you identify these are the actual approval patterns where you can ask business to repeat itself over and over again. Once that's done, then you can go back, update your comm, which is your cloud operating model. Because of this new service coming in, let's focus on this new service. Who's going to own this? How are you going to support this? Or how are you going to run books? So all these are actually tied in together. And then, of course, with the platform engineering team, now that we have a blueprint, work with the platform engineering team and say, okay, is there any blockers or area where you're concerned so that we can build these processes? So again, a lot of these things that we're talking ties in together closely. Yeah. No, I love that. Bringing in all of the little disciplines, security, finance, procurement, how are we going to make this thing move? It's like a balanced team, not just from the technical sense, but also from the business sense and the execution operational sense. I love it. That's some great tips. With that, I think we're in the last 30 seconds. So I want to just say I really appreciate the time. This is always fun when we get a chance to get together, and I'm sure we'll have an opportunity out in the lobby or something to spend some more time. Just want to say thank you for all your time. Hopefully this was... Anybody learn anything new here or here's some things that pushed the boundary for you? Okay, cool. Excellent. Then it's worth it, right? Well, thank you. All right. Thank you.
