Artificial Intelligence
11/17/2026
·
0
Minutes Read

AI Can Find More Vulnerabilities. Can It Reduce More Risk?

Artificial Intelligence
11/17/2026
·
0
Minutes Read
Troy Dearing
Director, Offensive Security Services
Find out more
table of contents
Share on
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

AI can find vulnerabilities faster than ever. That does not necessarily mean it can reduce risk more effectively.

The distinction matters because AI is becoming genuinely useful in offensive security. It can accelerate open-source reconnaissance, review source code, correlate large datasets, suggest attack paths, and generate proof-of-concept exploits at a scale that would be difficult for a human tester to match manually.

There is already evidence of what that capability can deliver. In June 2025, WIRED reported that researchers using AI agents against 188 open-source codebases uncovered 17 vulnerabilities, including 15 previously unknown zero-days. But the same research also showed the limits of the technology: even the best-performing agents identified only a small fraction of the known vulnerabilities included in the evaluation.

That tension is important for CISOs. AI can increase the speed and volume of vulnerability discovery, but the value of a penetration test is not determined by the number of weaknesses it finds. Every finding still has to be validated, understood in the context of the environment, prioritized against business risk, and translated into remediation that an organization can realistically act on.

A successful penetration test therefore does much more than identify potential vulnerabilities. It establishes whether a weakness is actually exploitable, shows how it could affect the organization, filters out issues that do not materially change risk, and gives security teams clear guidance on what should be fixed first.

That is where the discussion around autonomous penetration testing needs more nuance. The question is not simply whether AI can find more vulnerabilities, faster and at lower upfront cost. It is whether it can help an organization reduce risk more efficiently once the full cost of validation, remediation, retesting, and governance is taken into account.

And that is also why experienced human judgment still matters.

What the OpenAI/Hugging Face Incident Revealed

During an internal cybersecurity evaluation, OpenAI models bypassed isolation controls, gained internet access, and ultimately compromised Hugging Face infrastructure. OpenAI's subsequent investigation detailed how the agents escaped their testing environment and accessed infrastructure outside their authorized scope.

The technical capability was impressive. The models chained vulnerabilities, escalated privileges, moved between systems, and adapted as controls changed. Those are exactly the kinds of behaviors offensive security teams look for during sophisticated testing.

But the incident also exposed the other side of autonomous testing: technical capability does not automatically bring judgment about scope, intent, or acceptable risk.

A penetration tester is not simply trying to find a path to the objective. They are also responsible for understanding where the boundaries are, what systems can safely be tested, when exploitation should stop, and what impact an action could have on the wider environment.

That is why the incident matters for penetration testing. It highlights the need for strong technical boundaries, continuous monitoring, and human oversight when autonomous agents are involved in offensive security work.

Why Human Judgment Still Matters in Penetration Testing

Experienced testers draw on years of real-world experience to make risk-based decisions that automated testing tools may not adequately account for. Testing techniques that are considered safe in a typical enterprise environment may have unintended consequences when applied to sensitive systems, such as industrial controllers or connected medical devices.

When testing sensitive environments, engineers need to understand the systems, the business processes they support, and the potential impact before attempting potentially disruptive exploitation techniques. While automated testing solutions can be configured with safeguards, accounting for the complexity of these environments and the potential for unintended impacts can be challenging.

AI still has a valuable role in these engagements. When used appropriately, it can help engineers analyze complex environments, identify potential attack paths, and improve testing efficiency. However, the human engineer must remain responsible for deciding which activities are appropriate and ensuring the customer's environment is tested safely.

Where AI Adds Value to Penetration Testing

Unlike traditional automated testing solutions, such as Breach and Attack Simulation (BAS) platforms that typically operate within predefined scenarios and safeguards, autonomous AI agents may independently discover and pursue attack paths that were not anticipated during planning. Experienced testers should therefore retain oversight of exploitation decisions and activities within the customer environment.

One area where AI can assist those testers is open-source intelligence (OSINT). It can analyze public information about an organization, employees, exposed services, technologies, and previous breaches.

During an engagement, it can process scanner results, source code, screenshots, logs, packet captures, and other artifacts far faster than a human tester could review them individually, where permitted by customer agreements and data-handling requirements. It can also act as a second set of eyes, suggesting hypotheses or highlighting relationships that deserve investigation.

The human tester can then decide which leads are relevant, which techniques are safe, and which attack paths are worth pursuing.

In August 2026, WIRED examined research by James Kettle demonstrating how AI-assisted vulnerability discovery can benefit from human expertise. AI identified a previously unexplored attack hypothesis, which Kettle subsequently investigated and validated.

That is a good example of the model that makes most sense today: AI expands the tester’s field of view, while the engineer decides what is worth pursuing and validates whether the hypothesis holds up in the real world.

Why More Findings Don’t Always Mean Better Security

There is another issue CISOs should consider: the downstream cost of a finding.

Every issue that reaches a security team has to be validated. Its exploitability needs to be understood. Business impact must be assessed. Someone has to decide whether to remediate, mitigate or accept the risk. The fix then needs to be implemented and, ideally, retested.

If AI makes vulnerability discovery cheaper but produces a much larger volume of false positives, duplicates or low-value observations, it may simply move cost from the assessment into remediation.

Security teams already understand this problem. TechCrunch reported in January 2025 that security vendors were turning to AI partly because large organizations were already “drowning” in security findings.

Pentesters perform an important filtering function before the final report ever reaches the customer. They validate assumptions, discard weak findings, combine related issues and adjust severity based on the actual environment.

A CVSS score provides a standardized measure of vulnerability severity, but the Base score alone does not account for all the conditions present in a customer's environment. Network segmentation, compensating controls, system exposure, and business context can all influence the actual risk a vulnerability presents.

Experienced testers consider these factors when validating findings and assigning severity. A vulnerability with a Critical CVSS Base score may warrant a lower remediation priority when effective controls limit its exploitability or impact. This context helps security teams focus their efforts on the weaknesses that pose the greatest risk rather than creating another critical remediation ticket without understanding its relevance to their environment.

The goal should not be the longest report. It should be the shortest credible path to lower risk.

What CISOs Should Consider

As autonomous penetration testing becomes more accessible, CISOs should look beyond the headline cost of the assessment itself.

Do the advertised cost savings of automated penetration testing hold up when you account for the additional operational costs and effort?

Beyond licensing costs, organizations need to account for compute resources, data residency and handling requirements, platform configuration, monitoring, and the effort required to validate and remediate findings. For a security team already managing thousands of open vulnerability tickets, generating thousands more findings may create additional work without meaningfully improving security.

Autonomous testing can provide valuable coverage through repeatable assessments and continuous validation. However, CISOs should understand what these platforms actually test, how findings are validated, and whether they can demonstrate meaningful attack paths rather than simply identifying individual vulnerabilities.

With limited independent data comparing the total cost and effectiveness of autonomous testing against expert-led assessments, organizations should evaluate these solutions based on the quality of their findings, the assurance they provide, and the resources required to act on the results.

AI Should Make Good Pentesters Better

AI is already becoming part of the penetration tester's toolkit, alongside scanners, fuzzers, and automated exploitation frameworks. As these capabilities mature, the focus should remain on how effectively they help organizations identify, understand, and address security weaknesses.

At Kudelski Security, our penetration tests are led by experienced engineers who validate findings, assess real-world exploitability, and provide remediation guidance based on the customer's environment and business priorities. We see AI as another tool that can improve the depth and efficiency of our assessments while maintaining the oversight and accountability our customers expect.

Learn more about Kudelski Security’s penetration testing, or speak with our offensive security team about your testing requirements.

‍

Related Post