An Agent Is Not a Security Scanner Until It Can Prove the Bug

Oct 1, 2026 6 min read

The hard part of an AI security scanner is not finding suspicious code.

It is proving that the suspicion survives contact with a running system.

Google’s Product Security team says its internal PageBreak project has found more than 500 Cross-Site Scripting vulnerabilities across Google first-party web applications. The interesting detail is not the model name or the number of findings. It is the loop around the model: PageBreak does not send a plausible hypothesis to a product team as if it were a bug. It hands the hypothesis to a deterministic validator that tries to demonstrate the failure.

That is the difference between an agent that generates security-shaped text and an agent that can produce a high-confidence finding.

The problem is AI slop, not a lack of hypotheses

Google’s description of PageBreak starts with an operational problem. LLM-based scanners can identify complicated code flows, but they also produce noisy, unverified reports. A security team that receives a large pile of convincing guesses has not gained much. It has moved triage work from the scanner to the humans who own the application.

PageBreak was built to reduce that noise. Google says the project began as a pilot in November 2025, became a fully fledged project in January 2026, and is used mostly with Gemini models, including Gemini 3.1 Pro and Gemini 3.5 Flash. Those details matter, but they are not the architecture’s centre of gravity.

The architecture is a control loop:

The agent forms a vulnerability hypothesis.
A specialized validator receives the hypothesis.
The validator executes a real check against a running environment.
Only validated findings move toward the product team.
Failed or incomplete attempts become feedback for later runs and validator development.

The model searches the space of possible attacks. The validator decides whether a candidate survives.

Deterministic validation closes the loop

Google describes validators that are specific to the vulnerability class and application surface. For an XSS hypothesis, a validator can inject a payload, load the resulting URL through a rendering harness, and check whether JavaScript actually executes. For SSRF, it can look for a request to an internal service. For path traversal, it can create a file in a controlled location and test whether the application can read it.

This is not merely a better prompt. It is a different trust boundary.

A language model can reason that a value appears to flow from an HTTP parameter to a JavaScript sink. That is useful. It is still not proof. The validator supplies the missing observation from the deployed or test environment.

The same pattern appears in Google’s examples from the PageBreak real-world findings. One cache poisoning issue involved an unconstrained path segment being inserted into a JavaScript response while being omitted from the cache key. Another XSS issue involved a signed redirect parameter. The agent did not stop at noticing that the final request needed a signature. It found a related authorization flow that generated a valid signature for the malicious value.

The report is interesting because the agent had to connect application behaviour across requests. But the system still needed a concrete exploit path before the finding became high confidence.

The verifier is the product

Security teams often describe agentic scanners as a model problem. Which model can follow the longest code path? Which model can reason over the most context? Which model can discover the most unusual vulnerability class?

Those questions matter, but they are downstream of a more basic one: what happens after the model makes a claim?

Google says PageBreak uses non-deterministic findings as seeds for deeper inspection, to guide validator development, or to expose missing environment access. Crucially, those unverified candidates are not sent to product teams as confirmed findings.

That is a sound division of labour. The model is allowed to be creative in the search phase. The reporting phase is conservative.

This is the same design rule that reliable coding agents need. Exploration can be broad. State-changing actions and final claims need gates. An agent can suggest a patch, but tests and review decide whether it is acceptable. An agent can suggest a vulnerability, but a controlled reproduction decides whether it is real.

The verifier is not an accessory to the agent. It is the part that makes the output operationally trustworthy.

The uncomfortable constraint is environment access

PageBreak’s results also show why a security agent cannot be evaluated as an isolated model. Google points to three infrastructure advantages: a large monorepo with complete execution paths, security signals extracted from live HTTP traffic, and mature scanning infrastructure that can authenticate to many internal applications.

That context changes what the agent can see and what it can test.

A model with excellent reasoning but no authenticated access, no reliable request replay, no source-to-route mapping, and no safe validation harness will produce a different class of result. It may still find useful hypotheses. It cannot prove as much.

This is why benchmark scores alone are a poor proxy for an agentic security system. The system’s effective capability is closer to:

useful findings = hypotheses × environment access × validator coverage

If validator coverage is weak, a more capable model mostly produces more candidates for humans to reject. If environment access is incomplete, the system may miss the execution path entirely. If the control loop is sound, model improvements can compound because better hypotheses are fed into a reliable proof stage.

What operators should build

The PageBreak approach suggests a practical order of operations for teams building security agents.

Separate discovery from confirmation

Let the agent explore code, traffic, configuration, and unusual request sequences. Do not make the same unconstrained model responsible for declaring its own hypothesis true.

Give each vulnerability class a real validator

A generic “does this look exploitable?” score is not enough. Build controlled checks for the classes that matter to the application. A validator should state what it observes, what counts as success, and what access it requires.

Treat missing proof as useful output

An unconfirmed finding is not necessarily worthless. It may identify a missing validator, a missing credential, or an inaccessible service boundary. Keep it as a seed, but keep it out of the high-confidence queue.

Make the test environment safe by construction

Validators may execute payloads, trigger requests, write files, or interact with authenticated systems. They need isolation, least privilege, approval boundaries, and audit logs. Deterministic proof is not permission to let an agent run arbitrary exploit logic against production.

Measure false positives and false negatives separately

Google explicitly notes that deterministic validators can create false negatives when coverage is incomplete. A near-zero false positive rate is valuable, but it does not mean the scanner sees everything. Operators need both metrics, plus a record of which cases were not validated and why.

The model is only one component

Google’s PageBreak announcement is easy to read as another demonstration of a powerful security model. That misses the durable lesson.

The model proposes paths. Specialized tooling tests them. Infrastructure supplies the missing context. A policy decides what is allowed. A human or downstream system receives only the result that passed the required gate.

That design is less glamorous than a claim that an agent autonomously found hundreds of bugs. It is also more useful.

An AI security scanner becomes reliable when it can prove the bug, not when it can describe one convincingly. The next generation of agent systems will be separated less by who can generate the most hypotheses and more by who built the strongest verifier around them.