How do you plan for AI system failure?

Maria Haight: It doesn't mean you have to be on the bleeding edge for everything, but just listen and learn and watch. It does feel overwhelming some days. I get that as a CISO. And my head spins certain days, and I think, "My gosh. How am I ever going to manage this for our company?" So don't feel like you're behind. I think there's plenty of time to still jump in, learn, listen, and be willing to share your stories, too, because that's how we all grow together.

Johnny Hand: Welcome to AI Security Brief, where we're unpacking emerging AI threats, vulnerability research, and the strategic decisions that security leaders are making right now. I'm Johnny Hand.

Dustin Childs: And I'm Dustin Childs. This episode, we look at the readiness gap that opens after an AI initiative receives governance approval. Our guest today talks about how that approval and initial risk assessment are really the first half of the job. We still have the operational questions that follow.

Johnny Hand: Who monitors the capability? What happens when it fails? And can the organization recover the process that it replaced? Yeah, I'm really excited about this episode because Maria Haight is the Global Information Security Officer at Joyson Safety Systems. And I actually didn't get to record this episode with you because I was traveling overseas. But coincidentally, I ran into Maria a couple of days later at another event, had a great conversation around AI security and also some of the topics that you talked about. And for Maria she works in a very complex, highly regulated industry.

And when it comes to AI adoption, she has to account for not only the supply chain complexity, but regulatory obligations, operational resilience, and business continuity. She's also faced with a challenge that many security leaders are faced with right now, where the boardroom really wants to go fast on AI adoption, but we have to make sure that it's governed well, that there's security and that we're compliant in how we deploy these systems.

Dustin Childs: Yeah, Maria has a methodology for scoring this, but it's not a scorecard for any one company. It's a practical set of principles that security leaders can apply throughout their AI journey. I really enjoyed our conversation, and we touched on a whole lot of different topics. So let's go ahead and kick things off. So welcome, Maria, to the AI Security Brief. When a governance board signs off on an AI initiative, everybody treats that as the finish line, but you're arguing that it's not. It's barely halftime, and the band is still warming up.

What's the operational work in the months that follow that most organizations aren't planning for?

Maria Haight: Yeah, there's a number of things, Dustin. And what I first think about is even when we get approval, there really needs to be a very good understanding of that use case, as well as whether it's going to be a PoC, which is usually how we start off, because we want to make sure that this tool is doing what the audience expected it to do, that there's a time frame for that, and then there are monthly check-ins. So that's been a bit of a change for us: looking for a monthly report back on whether we're progressing in the way we thought.

Were the six months that we allocated you going to give you enough time to show the output of your use cases being positive or negative? And then really sticking to it. Sometimes those PoCs linger for periods of time beyond intention, and then we've lost why we came into this, and maybe we should be looking somewhere else now. So there's that. I would also say, too, one of the big things, even as you come in for approvals, is making sure we're not bringing in tools or processes that are redundant. I think that's a big thing.

I think a lot of companies are starting to lose this — the slide rule on that is that this sounds great, but, "Oh, we're doing that somewhere else, too. Why don't we piggyback you into them, and drive that forward?" So, okay, good question. Now, we got through all that. What do we need to be thinking about? One of the things that I think is really important is your data protection, for sure, and then it's the monitoring of that environment. If we're able to visualize utilization, great; users, great. But what about agents, and what data are they actually using and integrating into?

Everyone comes in with a use case. It doesn't always mean that's where we stay, right? Because new things are learned through the tool and through workflows and what the agent can do, and then it starts morphing. And that's where we have to keep an eye on things. So I think having good tools that give you the visibility into that growth is very key. And then there's the backup of that data. Any data that we're using to model needs to be backed up, right? Technology is not perfect. Things do go wrong.

And so we have to think about that piece, too, in case we have to start over.

Dustin Childs: Yeah, the backing up of the materials is very interesting. As someone who actually started his career on mainframes, where a large part of my job was hanging tapes for backups, it's near and dear to my heart. There's a human continuity risk, too, isn't there? I mean, are there people who remember the old manual process still around when the AI fails? I always go back to the TV show The Pitt. I don't know if you're familiar with it. In the second season, no spoilers, but there is a computer failure, and all the doctors and nurses have to go back to written forms.

And the young doctors are struggling to do anything without a computer in front of them. The old doctors are much more adept at adapting to the change. But how are we making sure that should the AI fail, there's a person who still remembers how to do these things? How do you plan for that loss of institutional knowledge?

Maria Haight: Yeah, that's been something that we've brought forth without having to learn the hard way yet. I've certainly heard some horror stories where AI got put into production and we're off and running successfully, and then all of a sudden the NVIDIA chip fails or whatever is going on fails, and now we're dead in the water. And if that is production-ready right now, you've got a serious business consideration to get around. And where are those people, and do they even remember the process?

So we've been developing lessons learned around each of our tools and how we did things and why we did it so that we could map it back if we needed to go back to a manual process. And so that's almost sitting as documentation alongside our agents today. But that's been our road. I think having good processes documented to begin with is always a good thing, regardless, because that may even help you with your better AI development. If you're just going off the cuff, right, you may already be missing things or forgetting things.

So I think understanding that process up front is always a good thing for any of us, but even more so if you're going to turn it into an AI capability.

Dustin Childs: On this podcast, we talk about how security leaders should deal with their boards, and especially when you have a divided board. So let's paint the picture of a divided board where one member is pushing to accelerate AI and the others are insisting on caution and compliance first. How real is that dynamic, and where does the security leader end up standing in that, at least to be successful?

Maria Haight: Dustin, we actually see that through our governance board approval process, because you've got all the different hats in there. You've got legal. You've got compliance. You've got data security. You've got InfoSec. You've got IT. And so we all bring those lenses to it. We definitely have technology people within our company that are really pushing us, and thank goodness for that. And I would definitely say our leadership wants us to push AI first to find efficiencies, to be able to get better product out, higher quality product out. But I think even when we're going through the approval process, we're pushing each other a little bit.

Something might come up, such as data, right? That's just the easiest one. What data are we talking about? And can we use that data in this way? You've got to look at your contracts. If you are looking to use customer data in some capacity in AI, some customers are saying, "I don't want you to do that." And so you've got to be careful if this is what you're planning to do and how you're going to do it, and that you make sure that they're involved in it. I think the next thing is certainly compliance in HR from a perspective of PII data.

That's a big one, right? Especially if you're under GDPR and what we're going to do with that PII, because right now, we've really tried to hold a line and say, "No PII data is allowed." The sensitive data or confidential data has dependencies on how it's going to be used and who will receive the data from the agent. So I think we've tried to not be that group because usually IT or InfoSec is the office of no. We've tried not to be the group of no. It's the group of how to.

Or, "We really don't want this, but can we do it this way?" Or, "Could we sample the data first? Show us the output. Show us how you're going to use it. And then we'd like you to come back again, and then we could talk about actually using the data that you're looking for that we thought maybe would be too sensitive for this particular use case." So trying to not be that no stamp, and instead the how-to stamp.

Dustin Childs: You spoke about this a little already, but I really wanted to bring it up because I think this is a very interesting thing that a lot of people don't consider, especially if you're not in that industry. And that's the heavy regulatory load. It can be its own force. How does the compliance burden shape not just whether you adopt AI, but how you architect the monitoring and controls around it?

Maria Haight: We put a very strong AI acceptable use policy together to make it very clear from the beginning what those guardrails look like. And that's what we lean on. So there should be very little speculation around that. That helps us, I think, to adhere to compliance, HR, and InfoSec needs and wants from any AI initiative from the get-go. We've trained, Dustin, pretty heavily on that, and we did ask the entire company, not just those power users, to read it and understand it and bring questions to us if they didn't and continue to do so.

We also then make sure as we give people license access to any of our tools that they also get trained on those tools. So again, how to use it in the way it was intended, without trying to circumvent anything the policy does not want us to do. And I really think that's been the key, is the constant communication and training so people don't have to guess. Like, we just tell them upfront, "Yes. No." It can sometimes be that black and white.

And I know it does not mean that there are no questions, but we want to give direct answers to those questions to help our staff.

Dustin Childs: Great. Well, now, Maria, you have a framework, right? The top things people forget when doing an AI risk assessment. Walk us through part of that list.

Maria Haight: When we do an AI InfoSec assessment, it really talks a lot not just about the architecture. It is also talking about data flows, accessibility, third-party utilization. We even go into some ethical discussions, too, right? We have to make sure that doesn't get lost along the way. Then we talk about the backup of data, the monitoring, the ownership. We do have a business owner for every one of our tools that we have approved so that they become the voice when we have questions. We don't want to be running around and having a lack of accountability.

If this person brings forward a tool for utilization, they then are the ones that are going to have to answer questions or be responsible if something is found that needs to be dealt with, such as we didn't put a business continuity plan together. Well, we need that. We need that because now, you're getting ready to go into production. Before we allow you to go into production, we want to make sure you've thought through that. And again, that your backups are ready, and that we have people able to support you, too. This doesn't just all fall in IT's lap. IT didn't build these agents.

And so you can't just walk away and assume IT is going to run in due to a help desk ticket, right? And then I think the other thing is the transparency of whether this is being successful or not. That has to be on the business owner, too. And that's critical, I think, even from an analysis, is that there's that due diligence that's being done. Otherwise, we need to reroute our time, our resources, our money to other solutions as well. So it really is that full suite, Dustin. I'm going to go back to that application model.

Any business owner that brings an application to IT, we've got to think through all those things, and this is no different in my mind.

Dustin Childs: Right. Yeah, I'm seeing that a lot of things when it comes to AI and governance, deployment, securing, whatever, are the same things that we've been doing if we've been doing security well. It's not that it's a new policy.

Maria Haight: That's right.

Dustin Childs: We just need to apply them to AI in a way that makes sense.

Maria Haight: That's right.

Dustin Childs: Right?

Maria Haight: Agreed. A hundred percent. Yep.

Dustin Childs: So of those things that are in your framework, what do you think would make the biggest difference for one of our listeners to fix first and why?

Maria Haight: I would ask you to start backwards, not at the beginning of development, but think backwards. Think in case there was an actual problem with your agent. Do I know where the support is coming from? Do I have a backup plan? Do I have a data plan for fallback if I need to rebuild that model in some way? It really is that good BCP/DR plan. Start there before you go into production. Now, I know that that shouldn't come before your development. Do your development, absolutely.

And now, we have something that we're ready to go mainstream with, and now I want you to go to the worst-case scenario. Because I think that too often happens after we've gone live and a hiccup happens, and now we're scurrying around. So I think that needs to be the lever to pull before you're ready to say, "Help us go to production."

Dustin Childs: So find your black swans before you go off flying.

Maria Haight: Find your black swans. Yes, perfectly said. I want to throw in one other topic, Dustin, if you don't mind. And I didn't think about it earlier, but I'm thinking about it now. I think companies need to make sure, as they make contracts with LLM providers, that they truly understand which of their data could be used for modeling and how far that goes. And there is a legal understanding that you should hold with your LLM provider regarding your data, even if it's metadata, where it's being stored, how it's being stored, and how it's being utilized.

I think people need to dig in a little deeper sometimes and have those conversations. And I would just press again on anybody to make sure, because you don't want data leakage. This is the worst way to go: someone pulls up a browser and goes into their LLM of choice, and all of a sudden they're just starting to see your information that was not supposed to be made public show up in the responses, right? So I think those are big conversations to have with your LLM broker of how that data is being utilized, maintained. Hopefully the retention period is something that's palatable for you.

I just don't know if enough people know to have those conversations.

Dustin Childs: Yeah, that's a very good point. And we've seen, in the past, how not having those conversations can lead to some major issues when it comes to personal privacy or intellectual property. So absolutely, very good point there to have those conversations with your LLM provider.

Maria Haight: Yeah.

Dustin Childs: What's one thing you want our audience to think about or do after they hear this conversation?

Maria Haight: I want them to go back to those that are building agents within their environment and make sure that they are looking at it with an understanding of scope, data utilization, and the intended use. If it's going to go external, you have a lot more things to be thinking about than if it's staying internal for a small population to use. Make sure your licensing is understood. And the one thing that I have learned through some podcasts that I've listened to, too, is a token is not a token is not a token, right? So the cost of a token and how that is created is very unique.

Inputs and outputs, data uploading. So don't get shocked, right? Make sure you are asking those questions, too, of your LLM providers of how that token is truly created so that you get a bill that's expected versus a shocker. Otherwise, now you've started having to change strategy really quickly because you didn't anticipate that was going to be the burn rate that you were going at.

Dustin Childs: Yeah, tokens used to just be for the arcade and Chuck E. Cheese, and now that's a very hot topic.

Maria Haight: That's right.

Dustin Childs: Yeah, I've even heard some people jokingly say, "we can donate our PTO. Can somebody donate tokens to us?"

Maria Haight: There you go.

Dustin Childs: I think it's great. So, Maria, one thing I think about when it comes to governance and deploying all these things is you have to educate users on how to do this, but at the same time, AI is changing at such a fast pace. There aren't exactly books on AI security because by the time you publish it, it's out of date, right?

Maria Haight: Absolutely.

Dustin Childs: So, where does education fit into your factor? And not only just educating users on what they're doing with the AI agents, but educating your security team on how to interact with this new technology.

Maria Haight: Yeah, that's a great point. And it's certainly becoming a much larger emphasis in security: how to use AI to improve that. We're very Microsoft-focused, and our team is using Copilot Security as their AI tool of choice at the moment. And so they're becoming familiar with it. Again, I've got to tell you, I still think that there's got to be some validation. So we can't just take those answers as is. We've got to use that as one potential place for analysis and triage, but it shouldn't be your only one.

And again, I think that you have the opportunity to hopefully improve it with you giving it some data that may make it better for the next time that situation comes up. So there is some of that. Our SOC team, I think, is comfortable, but I don't know that I've trained them as well as I could. And that's probably something a lot of us could look at. I think we get focused, Dustin, on the tools and not so much on the training of how to best use those tools. So it's a very valid point.

I think we have focused ourselves from a training perspective more on the users and specifically users that we've given access to tools to make sure they knew how to use the tool. And it's something our AIGB just talked about recently. Even the tool that we have used mostly within our environment, we've done tremendous first-round training, but we realize now there needs to be a second round of training. Now, I'm past that, right?

And now, I'm doing these other things, and we want to come back and help facilitate a little bit more knowledge there, as well as all of these tools keep having more features and functionalities. And I am sure our audience is not reading those release notes. And so do we, again, have ownership on that? And I said, again, that's why we have to have those tool business owners. Those business owners have to own that. It can't just be an IT thing or an InfoSec thing. We really need them to know their product well and continue to educate as well. But I would say you're right, Dustin.

From a security perspective, there should be more there. And I'm not sure I know what that looks like at the moment, other than even us getting to understand better how to use the tools that we have available to us today.

Dustin Childs: Maria, what's a piece of hard-won wisdom about AI security that you wish more CISOs would take seriously right now, even if it's not a popular opinion?

Maria Haight: Yeah. First of all, I'm going to go out on a limb, even though Mythos scared the crap out of all of us this year. I don't think this is something we have to be scared of. I really think this is something that we'll be able to utilize for our benefit. And I also think, too, it feels like we can't keep up at the moment, right? It's changing so fast. But I think with great networking, with lots of third parties such as yourselves putting podcasts out, there is a way to stay in the loop.

It doesn't mean you have to be on the bleeding edge for everything. But just listen and learn and watch. It does feel overwhelming some days. I get that as a CISO. And my head spins certain days, and I think, "My gosh. How am I ever going to manage this for our company?" But I do think we're all in the same boat together and that there is opportunity. And just talk and reach out. And I think you're going to find, more often than not, most of the people that you are talking with are feeling the same rub, and they want to help each other.

So don't feel like you're behind. I think there's plenty of time to still jump in, learn, listen, and be willing to share your stories, too, because that's how we all grow together.

Dustin Childs: And, Maria, I want to thank you for a great conversation today. I really enjoyed it, and we really appreciate you sharing your insights and the full picture of AI security, and can't wait to have you back on the show sometime in the future.

Maria Haight: Thank you, Dustin. Appreciate your time. Have a great day.

Dustin Childs: Well, Johnny, I really enjoyed that conversation with Maria. What are your thoughts?

Johnny Hand: Oh, man, I loved the episode, to be honest with you. And many of these things that you guys talked about are exactly what I got to talk about whenever I was with her at that event. So it's really good reinforcement. I do like the honesty that we're talking about here, that once you start a project, right, and you're trying to get approval for AI governance, that that's just the starting point, right? It's not a proof of operational readiness. It's not a proof of impact. Also, when it comes to the capabilities, if you're developing these AI capabilities, you have to have an ownership aspect of this.

And especially if you're replacing those human processes, right? So now, you have to look at monitoring. You have to look at disaster recovery and business continuity responsibilities. You can't just set it and forget it.

Dustin Childs: Yeah, Maria really talked about the contingency plan that must protect more than the AI model itself, and I really agree with that. You have to preserve the source and training materials the capability depends on, along with the knowledge needed to validate, rebuild, or replace it. And you have to ask yourself, "Are the people who really understood the original manual process, are they still available if the AI system becomes unavailable or unreliable? Are you going to be left in a situation like 'Space Cowboys,' where you're calling in your retired senior citizens who are the only ones who understand how the old process worked?"

Johnny Hand: That's a great reference. I also think that there's something more to operational readiness, right? Whenever the boardroom says "Go," right, it's not done. There's a lot left to be done there. And we have to translate that as leaders between how do we move quickly but also have compliance and security in those aspects. And with Maria she works in a very complex industry. She has a lot of third-party dependencies and business impact downline. So I think her approach to how she's putting in that contingency planning is really, really smart.

Dustin Childs: Yeah. I love the practical next step that she provides, which is to revisit the AI risk assessment after approval and identify the operational step the team is most likely to have missed. You need to confirm who's monitoring the ownership, recovery, continuity, data preservation, and human fallback before scaling the capability. And I think it's great she emphasizes this, and I totally agree with her. This should be an ongoing readiness discipline rather than a one-time governance episode. So I love exactly all the points that she hit.

Johnny Hand: And that does it for another episode of AI Security Brief. We want to thank you for joining us, and our goal is always to host conversations that get you thinking differently about security. And if it does, consider subscribing so you don't miss what's next.

Dustin Childs: AI Security Brief is mixed and produced by Elliott Peltzman, with original music by Amneajynx. Our show producers are Melany Gallant and Alyssa Gonneau. Our executive producer is Jennifer Eiben, with content strategy by Ma'ayan Plaut. Additional production help by Liz Stokes. Video editing by Sarelle Joppy and Brigitte Criqui-Wild.

Johnny Hand: Thank you so much for listening. We'll see you next time on the AI Security Brief. [Music]