The harness matters more than the model

Ask a vendor why their scanner found a bug and the answer is usually the model. Ask why it missed one and the answer is usually also the model. Neither answer is complete, and the incomplete part is the interesting one.

What a model is given

A model does not audit a repository. A harness does, using the model. The harness decides which files the model may read and in what order, which tools it can call, how many times it may try, what counts as a finding, what counts as proof, and when to stop. Every one of those decisions is written down somewhere — in a system prompt, a tool manifest, a loop with a budget — and every one of them changes the outcome.

Give the same model a harness that reads only the files touched by a pull request and it will find the bug in the diff and nothing else. Give it a harness that traces every route from the entry point and it will find the authorisation gap three files away, and it will cost more to do it. Give it a harness that demands a reproducing input before it may report anything and its precision goes up while its recall goes down, and you will argue less with its reports. None of that is the model changing. It is the instructions changing.

This is not a new observation. Aikido Security made the same argument earlier this year in a post titled "How Aikido finds more vulnerabilities than Mythos at half the cost", whose conclusion was that the harness still makes the difference. We agree with the argument and draw a different conclusion from it.

If the harness is what matters, it should be open

A harness is text. A manifest and a set of instructions. Text is the thing software teams already know how to handle: it goes in a repository, it has a commit hash, it gets diffed and reviewed, it gets versioned, it can be forked when it is wrong.

Closed scanners treat the harness as the product and hide it. That is a reasonable business decision and a poor engineering one, because it leaves the customer with the one question that cannot be answered from the outside: what did it actually do? A finding without its trace is an opinion. A missed bug without the log is a mystery.

Midkernel's position is that the harness is the thing to publish. Every workflow in the registry is a public folder: a YAML manifest that declares tools, scope, models and budgets, and Markdown instructions that say how to audit and what counts as proof. Every run pins the commit of the workflow it used. If the workflow made a bad call, you can read the call.

If the harness is what matters, it should be scored

The second half of the argument is that a claim about a harness should be checkable. Vendor benchmarks are run by vendors, on tasks chosen by vendors, with results summarised by vendors. That is not an accusation; it is just where the incentives sit.

The bench exists to move the incentives. Every workflow in the registry runs on public datasets — CyberGym and EVMbench among them — in the same sandbox a customer gets, with the methodology on a public page and the logs behind every cell downloadable. Scores are published as they land, and disputed in public when someone disagrees. The point is not to declare a winner. It is to make "which one should I run on this?" a question with an answer you did not have to take on faith.

What this means for you

If you run one AI scanner today, you are running one harness, and you cannot see it. The practical version of this post's argument is small: read the harness before you trust its output, and compare two on your own code before you buy either. Both of those are the point of Midkernel once hosted runs ship.

The model will keep getting better. So will the harness — faster, because more people can work on it.