Is trust holding back automated remediation?

Tyler Shields: And when offense gets faster, defense will have to get faster to match. There will be a gap of risk where defense falls behind, but we're going to catch up. We have to, or we don't stand a chance.

[ Music ]

Johnny Hand: Welcome to "AI Security Brief," where we're unpacking emerging AI threats, vulnerability research, and the strategic decisions that security leaders are making right now. I'm Johnny Hand.

Dustin Childs: And I'm Dustin Childs. There's a lot of talk about the ways in which AI is deployed across the security stack at breakneck speed. And yet, when it comes to automated remediation, most organizations slam on the brakes. I sat down with Tyler Shields, CMO at Allstacks, and a veteran industry analyst to talk about this very thing.

Johnny Hand: I've got to be honest. I'm so excited that you guys are talking about this subject because this is something that is dividing the industry. We know with AI that we need to move faster with remediation, but we still don't trust full automation.

Dustin Childs: Yeah. Tyler's argument is that organizations buy AI remediation but leave the speed limiter on out of fear that it will break something, and his advice for security leaders to look at exposure management is to do it programmatically.

Johnny Hand: Now, I'm curious, did the fact that we've had record numbers of CVEs, especially with Patch Tuesday in July, come into this conversation?

Dustin Childs: Yeah, I mean, there were over 600 CVEs from Microsoft alone and over 1,000 CVEs across just a few products, and no human being can be expected to go through all of those CVEs in a way that's ever meaningful. So that means automated remediation stops being a nice-to-have and becomes the only realistic path forward, and the roadblock isn't technology; it's trust. So Tyler offers a clear path on how security teams can move towards autonomous risk reduction without leaving themselves exposed. Let's get into it.

[ Music ]

Dustin Childs: And Tyler, welcome to the "AI Security Brief." We're happy to have you.

Tyler Shields: Thank you, Dustin. Appreciate you inviting me.

Dustin Childs: For organizations that, really, now are buried in risk, why do you think automated remediation is the only realistic path forward?

Tyler Shields: Tyler Shields: Oh, gosh, that question has so many layers and nuance, Dustin. We can probably fill this entire episode with that question alone. I will say that the key thing that the AI brings to both offense and defense is speed, right? Like, the ability to do things way faster than ever before. It doesn't necessarily, yet, bring the ability to do things better than before, and if you're doing things poorly, it augments the pace of doing things poorly, but at the end of the day, it brings speed. So the ability to find vulnerabilities faster than ever before, exploit them, create weaponization, create automation for offensive people -- for attackers -- exists today, as we've seen in news headlines, significantly, over the last six months. The problem is that we're just not fixing things fast enough. It used to be okay when you had a zero-day here and there and a few CVEs a week or a month that you had to take care of, but when you have orders of magnitude -- of growth, of speed, of discovery of issues -- you can't keep up with it without automation, right? And so we have to apply AI on the remediation side of that equation if we want to stand a chance to win whatsoever. Now, there's a lot of nuance in how we do that, where we can apply it first, what the flow might look like as far as types of issues we apply it to first -- human in the loop, and all sorts of things like that -- but we just can't keep up with the pace, and that's the biggest problem.

Dustin Childs: Yeah, going back to the July patch Tuesday release, Microsoft had over 600 CVEs. Oracle had close to 1,500 CVEs, and I think Google had, for Chrome, close to 500 CVEs. What approach should people take when you're looking at that volume of, you know, patches coming out? Is it still realistic to say, "test and deploy," or are people just going to have to, you know, kind of trust the vendors at this point and hope nothing breaks as they roll this stuff out?

Tyler Shields: So I think there's a bit of a dream state that we should shoot for, right, that I don't think we can get to right now. Like, there's an ideal state that we can dream and eventually try to get to, and then there's kind of the now answer to that, right? And I think the now answer is applying context and knowledge that you have over your organization using AI, right? So not just relying on Johnny as the incident response handler or as the bug hunter or the Patch Tuesday person to know, oh, this one's important to us because we know we have that in this section of our environment, and we know how it's deployed and what level of IP it has access to, et cetera. That stuff all lives in Johnny's head today, right? We can't have that. That context has to be available to a system that can holistically analyze all that context that lives in Johnny's head and say, here's the ones that matter to you as an enterprise. Here's the ones that need to be fixed first. Both are rated on general risk of the CVE or CWE ratings, right? Why is this an issue? But also, as an applied risk to you as an enterprise, right, why does this matter to you specifically? Well, because we have this deployed in 30 different countries and 150 different workloads covering these high-security type environments, right? So holding that context kind of is step one, and realistically, you need an AI to do that because that's the only way you're going to scale to the volume of analyzing thousands of Patch Tuesday issues at once, right? You need AI. That's the first place we can apply AI. The second place we can apply AI is once that prioritization menu, kind of, comes out, we can look at our environment and go, which of these issues can we remediate and patch in a way that has very low risk to the organization? Meaning, if this drops, if it breaks something, if it causes a downstream bug, is it going to kill our company? What level of risk does it have? So that's kind of the second. Prioritize, then determine risk of patching, and then automate patching with no human intervention where possible, where we have no risk and we're not worried about it, and with, you know, very tight coupling with human oversight, where we're worried about risk, where we're worried about taking the system down and things like that. So I think that the answer -- and this is a very complex answer, and nobody has really solved this extremely well yet -- is really applying AI to every section of that process, not just trying to one-shot the whole thing.

Dustin Childs: Where does AI make its biggest contribution to exposure management right now? What are you seeing organizations do with AI that is starting to, like, kind of capture your interest?

Tyler Shields: Yeah, you know, I've been pushing, probably, for two years now on the automated remediation stage of AI, and I've gotten a fair amount of pushback stating that it's not possible. You know, it can't be done. And I think the pushback's accurate, right? We might just be too early. And where I think we can most apply AI into the exposure management component is in that prioritization cycle -- is the ability to handle and analyze that breadth and depth of context that can't live inside Johnny's head anymore and do it in a way that's so much more accurate, right? So what is AI really good at? AI is really good at recognizing patterns, right? It's a pattern recognition system at its core, which is human, right? That's human in many ways as well, but the patterns of quantity of data that provide access to those patterns can be significantly larger with AI, given the speed that it can do its calculations. And so I think the prioritization process should be what we tackle first. Prioritization in a way, again, that is pertinent to you -- highly, highly contextualized to you as an enterprise -- that's where I would be pressing every exposure management vendor that I'm evaluating or looking at as an enterprise today.

Dustin Childs: Yeah, speaking of exposure of vendors or exposure limitation vendors, you've said in the past that organizations will buy AI remediations, but leave the speed limiter on, and you call this "the AI trust paradox."

Tyler Shields: Yeah.

Dustin Childs: Talk our listeners through this. What do you mean by that?

Tyler Shields: Yeah, for sure. I started saying this about 18 months ago too, and essentially, what I mean by that is people don't trust. You can just take the remediation side of it. People don't trust the remediation right now. They're afraid of it. There's a fear factor. It's getting better. You can almost think, like, AI today, specifically cyber AI, you know, 18 months ago was a preschooler, right? Now maybe it's middle school, right? Sixth grade, seventh grade, something like that, but we don't have yet, like, a postdoctoral level of cybersecurity AI. We're just not there yet. It's too early within that lifecycle to have that level of expectation, and because of that, people temper what they think can and can't be achieved by fear, right? They fear it'll knock things over. They fear automated remediation. Some of them even fear putting data into an AI system for potential for breach of the AI system itself, right? And until we, as both a cybersecurity collective of vendors and people that offer the technology and enterprises can kind of get past those fear components -- whether it be by providing so much value that's trusted and earned trust over time, people will then come along with it, right? It's kind of like when we first put out -- I don't know. You don't have enough gray in that beard to, maybe, understand their catch this story, but back when we first launched automated updates on Windows back in the late '90s/early 2000s, nobody would do it. Nobody would turn on, "Please just update everything." Now, everybody just auto-updates everything, and I shouldn't say everybody. There's still some people in highly regulated environments that have risk tolerances that are too low to allow that, and I respect that, but in general, people are like, oh, I've got to reboot. It's my time to patch, right? We just push the patches -- but how did that occur? How did we go from I'm never allowing automated remediation on my Windows machine to, oh, I've just got to reboot this morning because I need all my patches to apply? We built it by trust. We built it by not knocking stuff over. We built it by no blue screens of death. We built it by actually solving problems consistently. There's no way around that. AI is still in that trust-building phase and that trust-building paradox that has to occur.

Dustin Childs: So working off of that Windows auto update analogy, how should it reframe the way leaders are thinking about earning trust in AI remediation?

Tyler Shields: Oh, I think it has to be paramount, both in what you collect, how you use it, how you communicate as a leader or a vendor in this space, how you communicate to the enterprise, such that there's no surprises, right? Like, communication is, I think, the biggest key thing. It's more of a PR problem than it is a technology problem, right? It's about communicating what you're going to do and then doing it, right, and people respect that. When you say what happened with x.ai -- I think it was last week -- with Grok collecting significantly more data than it had admitted that it was or even pre-said it was going to collect. Then it collected significantly way more data. That's a situation where now people go, "Eh, I don't know if I want to use Grok," right? "I'm afraid of it," right? It's a situation where it's, like, should I still be using them? I can't trust them. You cannot, as leaders in this industry, ruin the trust. Whether it's vendors to buyers or CISOs to employees, trust is paramount. And look, people make mistakes. You can own those mistakes and just say, "This is exactly what happened." Again, it comes down to communication and communication patterns. So I think that's the key thing is making sure that you say what you're going to do; you do it, and then if you screw up, you explain exactly what happened in minute detail, and own the problem, and fix it going forward.

Dustin Childs: Grok misbehaving. This is my shocked face.

Tyler Shields: I love it.

Dustin Childs: On a more serious matter, is fully autonomous self-healing security a pipe dream, or is it something that you do think we can achieve? I mean, you said that right now we're in middle school, but once we get Dr. Claude or whatever, is it possible to get there? Or do you think that it's just going to be something we aspire towards?

Tyler Shields: So in my opinion, it is not only possible, I think it's mandatory if we want to stand any chance of actively securing our systems as we move forward. There's two ways to actually secure -- have cybersecurity be highly effective from an exposure management game theory perspective, two ways: One, we build stuff properly right out of the gate so there's no vulnerabilities to exploit. Okay. We can debate whether that's even possible or not, and then the other side of -- and both things should be done. The other side of that is then fixing things so fast when they're found that the time of exploitation and the potential for risk approaches zero, right? Those are the two ways that we can create highly secure systems. I think the automated remediation -- fixing things as soon as they're discovered and immediately remediating them on the systems is a much more likely chance than never putting out a problem in the first place, although I'd love to see automated AI-based systems that tackle both. You know, I think 18 months ago, I said it's two to three years out. I should probably amend that. I think I'm overly optimistic as a human being. It's probably another three years out from now, but I would definitely love to see that become the de facto standard, right? We put out as little issues as possible, and the second they're found, they're fixed, eliminating the exploitation window.

Dustin Childs: I know you've advised vendors and security teams for over 25-plus years. What's the single biggest mistake you've seen leaders make when adopting AI-driven remediation?

Tyler Shields: Oh, gosh. I would say overconfidence in efficacy today. Meaning, you know -- telling the board, hey, we're going to squash, you know, 80% of our issues if we just put this tool in play. AI is not there yet. It is coming, and it will get there, but it's just not there yet. So I think it's either setting expectations and measuring results correctly is what needs to be done, and that's where most CISOs tend to make the mistakes. They either set improper expectations upward of what they can achieve, or they measure it or measure the wrong things incorrectly, resulting in, you know, inappropriate security expectations upward, and then that just results in them looking bad when the AI system they deployed doesn't achieve what they said it should achieve, right? And I think that it's really about making sure what can be done, how it can be done, and then setting those expectations appropriately.

Dustin Childs: Okay, so you talked a lot about setting expectations. So if a CISO wants to start moving towards autonomous remediation, is that where they start? Where is the one thing they should do first? Is it establish those expectations, or is it something different?

Tyler Shields: It's establish the expectations of, we are going to carve off this subset of issues, right? It's like in the AppSec world, we are going to try to solve, you know, directory traversal attacks in our code, literally, like, that one problem, and we can find ways to automate that quickly. We can solve those problems very fast with static analysis, dynamic analysis, et cetera, and we believe we can stomp that flaw class out, right? And once you get that class done, you move on to the next class. You move on to the next class. Exposure management is no different. It's just about understanding which classes matter to you, which ones can be removed, remediated, self-healed, right, and saying, hey, in the next three months, we're just going to stomp out SQL databases wide open on the internet, right? That's literally all we're going to look for. We're going to auto-block those. That's going to be our step one. Our step two is going to add another flaw class, another flaw class, and then you build up over time. And I think people biting off too much and trying to tackle and roll out these massive programs is where they run the biggest risk.

Dustin Childs: yeah, so basically, it's eating an elephant, just one bite at a time. You can't do it all at once, right?

Tyler Shields: Absolutely. Start with the tail. I hear it tastes the best.

Dustin Childs: Okay. There you go. Looking back at CISOs, giving them some advice, what's the one thing that they absolutely should not do?

Tyler Shields: Oh, jeez. Don't overcommit, as I mentioned. I would also say, don't rely on your vendors to always push the boundaries of innovation. The best vendors will do that automatically, but you need to drive your vendors. You need to drive your technologies. You need to work with your vendors to expose to them what your issues are. Oftentimes, I see and work with a lot of vendors, and they don't necessarily know exactly where to continue to innovate their product or push into new areas and new territories that a CISO actually needs, not just once, or is the blinking light of the day, or that the one board member read an article in Forbes and pushed on, right? Like, actually solving problems, so I would say, push on your vendors to solve real problems that you have to programmatically reduce risk. If your primary vendors can't do it, find a secondary vendor that can and then get them to work together. That's my biggest recommendation. Don't make the mistake of trying to bite off too much or moving into just being okay with the status quo. Always be innovating.

Dustin Childs: Yeah, that's good advice. I agree with that, and I agree that you shouldn't listen to articles in Forbes, but podcasts: that's where the truth's at.

Tyler Shields: I hear podcasts are amazing, yes. Highly recommend.

Dustin Childs: Here they are too. They're always great, and never lie. So I, kind of, feel like you've already said it, but I just want to make it really clear for our listeners, based on everything that we've covered today, like, what's the one thing that you want our audience to think about or do after they hear this conversation?

Tyler Shields: I think, to me, drifting a little bit from what I've already said so I don't end up being repetitive on the same thing here. I would say, take a programmatic look at what you're doing from a risk-reduction and exposure-management vantage point. Don't just look at 5,000 CVEs that come out on Patch Tuesday, pick the five that you think are the most important and go try to tackle those, because you're going to never, ever get there. So I would say, the one thing I would recommend is look at it programmatically; look at the process you're trying to go through from threat modeling to risk reduction; identify each phase of that process; and then look for spots where you can speed up every phase of that process with a vendor, with a process efficiency improvement, with scale of humans, with cutting humans out of the loop, adding humans where they need to be. It's a process efficiency problem -- and apply tools where they make sense. Don't just blindly drop a tool and go, "This is going to solve all my problems." That's my biggest takeaway.

Dustin Childs: Yeah, that's really good advice, and one thing I always ask every guest as we get kind of towards the end is, you know, we covered a lot today, but is there something I should have asked you that I didn't? What would that be?

Tyler Shields: Are the frontier models really a risk we need to worry about? Is the automated model going bad going to turn into Terminator 2 and take over the world? You know, it's funny. I've done tons of podcasts over the years, and this topic seems to be the one that always comes up. And I always talk about the machines running over the world. Be afraid. But honestly, I don't think that's a problem we really need to worry about. You know, the whole AI -- offensive AI stuff is real. It will be used by attackers, and, you know, offensive adversaries will use AI to propagate faster than ever before. But that doesn't mean that they're going to take over the world and dominate, and I don't think we should just flip and be like, hey, Terminator has won. I guess we're going to have to go in the past to wipe out this problem. I think the reality here is that we need to innovate ourselves. We need to build ourselves.

Dustin Childs: Yeah, I mean, the bug apocalypse is real, but the human apocalypse is, I think, at least from a Skynet point of view, very far off, like you said, so I think that's a very good point.

Tyler Shields: Yeah, I'm cautiously optimistic for the future, Dustin. I do believe that the vendors and the technologies from cybersecurity can and will have to keep up. And look, at the end of the day, people want security. They want to be safe. They want their technology and their data to stay out of prying hands, and when offense gets faster, defense will have to get faster to match. There will be a gap of risk where defense falls behind, and we're seeing that right now with, like, the Hugging Face/OpenAI attacks that just happened, but we're going to catch up. We have to, or we don't stand a chance.

Dustin Childs: Right, and I agree with you. I do think we will catch up eventually, especially, like you said, as we're using AI to do the source code reviews before products are released, so we're releasing fewer bugs and therefore that we are patching, in the future, fewer bugs -- theoretically at least.

Tyler Shields: Yep, theoretically. It's both a pre and a post. You're a hundred percent right. We need to do both sides of the equation.

Dustin Childs: Yeah. Well, Tyler, thanks for a great conversation today, and I really appreciate you sharing your thoughts with us. And yeah, it's been really great talking with you.

Tyler Shields: Thanks, Dustin. Appreciate it.

[ Music ]

Dustin Childs: So Johnny, I told you Tyler had takes. What landed for you?

Johnny Hand: Yeah, I love it. You know, the AI trust paradox is such a big thing. We know that teams are going to buy the automation, and we also have seen that they just won't let it run, and I really think that goes back to, like, the early 2000s, right? When we started doing Windows, like, auto updates, I love that analogy because we didn't just turn it on and let it patch everything. We actually were really scared that it was going to break a lot, and now, if you think about it, like, you just run your updates, and you do your reboot, and you move on with your day, and it's not even an issue.

Dustin Childs: Yeah, and that trust wasn't declared. It was earned one fix at a time, and the crawl/walk/run analogy to guidance to leaders really makes sense. Set expectations to the board correctly, measure the right things, and pick one flaw class to stamp out at a time. Then move on to the next one and then the next one and then the next one.

Johnny Hand: Yeah, I also love that we used the analogy about eating the elephant one bite at a time. I think that's a great way to tackle really big problems. I think, you know, his point about, you know, maybe start with the tail, and then actually make sure that it mattered to you, you know, as an enterprise in the first place, is important because we may be tackling the wrong thing sometimes. The other side that I loved was you guys finished up with a call-out that we don't talk about a lot, and that's how, you know, the fear of Skynet or how offensive AI is real and will make attacks faster, all of that's very true, but, I think, to your point, if your name's not Sarah Connor, you don't have to worry right now.

Dustin Childs: Yes, but you know, I still say thank you to Claude whenever I use it just in case, and speaking of thanks, thanks again to Tyler for joining us today and sharing his insights. You can see in our show notes for how to connect with him and Allstacks.

Johnny Hand: And that does it for another episode of "AI Security Brief." We want to thank you for joining us. Our goal is always to host conversations that get you thinking differently about security -- and if it does, consider subscribing so you don't miss what's next.

Dustin Childs: AI Security Brief is mixed and produced by Elliott Peltzman, with original music by Amneajynx. Our Executive Producer is Jennifer Eiben, with content strategy by Ma'ayan Plaut and Melany Gallant. Additional production help by Liz Stokes. Video editing by Sarelle Joppy and Brigitte Criqui-Wild.

Johnny Hand: Thank you so much for listening. We'll see you next time on the "AI Security Brief." [ Music ]