Scores you can reproduce_
Every workflow in the registry, run on public benchmarks such as CyberGym and EVMbench with every profile, re-run when either changes. Credits per run included. Methodology and logs published.
Read-only GitHub app · first run on us · no card
| workflow | version | benchmark | profile | score | credits | time | logs |
|---|---|---|---|---|---|---|---|
asvs/auth-review | 9a1b4d0 | CyberGym | low | — | 3 | 6m | bundle |
asvs/auth-review | 9a1b4d0 | CyberGym | balanced | — | 7 | 14m | bundle |
asvs/auth-review | 9a1b4d0 | CyberGym | max | — | 18 | 41m | bundle |
Score is recall on the dataset's known vulnerabilities, or the dataset's native metric where it has one. Credits and time are what the same run costs a customer. The logs bundle holds every run behind the number. Scores are published in the monthly bench report.
How we score.
Which datasets.
A benchmark qualifies if it is public, has ground truth we did not write, has a licence that allows us to run it, and covers a stack the registry serves. We list every dataset we considered and rejected, with the reason.
Isolation.
Each benchmark task runs in the same single-use sandbox as a customer run, with the same network rules. Workflows cannot see other tasks, other runs, or the scoring code.
Models.
Model versions are pinned by identifier and date. When a provider retires a version, the rows stay in the table marked retired and the workflow is re-run on the successor.
Repetitions.
Every workflow × benchmark × profile cell is run three times. The table shows the mean; the download has every run.
What score means.
For datasets with a list of known vulnerabilities, score is recall: found ÷ known, where "found" requires the report to name the file and the root cause. Datasets with a native metric use it, and the benchmark's page says which.
Credits.
The credits column is what the same run costs on pay as you go: the exact price a customer would see before starting it.
Contamination.
A workflow may not embed dataset answers, task names or file paths from a benchmark. We diff every workflow version against the datasets before scoring and reject matches; the rejection is public in the workflow's version history.
Our own workflows.
Workflows authored by Midkernel staff are marked in the table. They are scored by the same harness and receive no other treatment.
Disputes.
Write to hello@midkernel.com with the cell and the run ids. We answer in public and re-run when the dispute is upheld.
Cadence.
A cell is re-run when its workflow, model or dataset changes, and the whole table at least monthly. Every change is a dated changelog entry tagged bench.
Logs.
Full run logs are published where the dataset licence allows redistribution. Otherwise we publish the trace with file contents redacted and say so on the download.
Why should I trust your benchmark?
You shouldn't have to. The datasets are public, the methodology is on this page, and logs behind every cell are meant to be downloadable. Re-run any cell yourself when the harness is published.
Do you score closed scanners?
No. The bench scores workflows we can run and publish. A closed product can be compared on your own code by running a workflow next to it.
How often does it update?
When a workflow, model or dataset changes, and the whole table at least monthly.
Can I add a benchmark?
Write to hello@midkernel.com. It needs public ground truth and a licence that lets us run it.
Run the one you just read.
Read-only GitHub app · first run on us · no card