Why we didn't build an AI pentester

Every few weeks now there's a story about a model finding a real vulnerability in real software, unaided. They're true, and they're going to keep coming. The reasonable question that follows is whether tools like this one have a future, and the answer we keep hearing from other vendors is to bolt a model onto the product and call it AI-powered security.

We went the other way. Draugr's core doesn't use inference at all, and won't. Here's the reasoning, because it's more interesting than the conclusion.

What AI commoditizes#

The thing models are getting good at is judgment on unfamiliar code. Reading a codebase nobody documented, noticing that two innocuous functions compose into something dangerous, chaining a foothold into an exploit. That's the pentester's craft, and it's expensive, scarce, and slow.

Making it cheap is a real change, and mostly a good one. But notice what it isn't. It isn't the job most security tooling does. Draugr doesn't discover novel vulnerability classes. It runs known checks against a known surface, every time you push, and tells you whether you can ship. Those are different products that happen to share a category name.

It is also worth separating producing candidates from finding vulnerabilities, because the first is easy and the second is the job. Pointed at one large Java codebase, two methods produced very different lists:

Neither number is an argument that static analysis beats models: CodeQL on its own confirmed nothing on any of the seven repositories either. Both halves failed at the same step, and the answer is to compose them rather than to pick one.

Four things a model can't give you#

Not "can't yet". These follow from what a language model is.

The first of those is the one with a measurement attached. Run the same scan on the same code five times and the findings sort themselves into two piles. This is the pile the model invented:

Roughly half of what it invented showed up in only one run of the five. The other pile is the findings that were genuinely there, and the model did far better: around 85% of those showed up in every run. It recognizes a known problem reliably and invents the rest unreliably, and a gate has no way to tell which pile a given finding came from.

A finding present in three runs out of five is one your pipeline waves through two times in five. And a false positive that moves cannot be suppressed, because there is no stable thing to suppress, which removes the one technique every team with a large backlog depends on.

The equilibrium those four imply:

Deterministic tools at the inner loop, AI at the outer loop.

We'd rather own the inner loop than pretend the distinction doesn't exist.

The risk worth planning for#

Here's the part the headlines miss, and the part we spent the most time on.

Our scanners were never the differentiated bit. Trivy, Semgrep and Gitleaks are excellent and free, and anyone can run them. What Draugr adds is the layer above, deciding which controls apply to which component, normalizing four tools' output into one schema, ranking findings by what they would cost you, and turning that into a verdict.

That layer is exactly the kind of glue an agent can improvise:

Run trivy and semgrep on this repo and tell me what matters.

That is a plausible prompt, and somebody is typing it right now. If that is all Draugr is, an assistant routes around us in a year.

So the honest question was never can AI replace scanners, but what do we have that an improviser can't produce, and the answer turned out to be the same thing in every case. something written down and reviewed.

Agents are strong at inference and weak at institutional memory. So we leaned into being the memory.

What we built instead#

If an assistant is going to answer security questions about your code, and it is whether or not anyone planned for that, the useful thing is for it to answer from your descriptor rather than from a scope it invented.

So Draugr speaks the Model Context Protocol. Your assistant asks Draugr which controls exist, what your Saga says, what the last scan found, and gets back the same prioritized answer your pipeline gets. A scan that only reads runs when the assistant asks for it; one that would probe a host, send data to a third party or deliver the report off the machine asks you first, with a prompt that names what it will do.

That's the position, deterministic core and AI at the edges.

The uncomfortable part#

Cheap autonomous exploitation is not good news, and we're not going to pretend our tool neutralizes it. Attackers get the same cost reduction defenders do, and probably use it more efficiently.

But the response to time-to-exploit collapsing isn't a smarter one-off assessment. It's closing the fundamentals continuously, on every commit, before the thing ships, which is a throughput problem, and throughput is what deterministic tooling is for.

Meanwhile AI is writing an enormous amount of code, held by teams with less security expertise per line than ever.

More software, faster, from people who've never configured a scanner.

Demand for boring, repeatable, hand-holding security tooling goes up in that world, not down.

We're betting on boring. Try it, which is Apache-2.0 and it runs entirely on your machine.