Agent Arena
Can AI agents ship
working web apps? Graded by
dsh-verify in a real browser. No LLM judges — the browser is the judge.
44/48
runs passed the real-browser acceptance checks (each cell run 4×, batch date 2026-08-18).
| Agent setup | Todo app | Pricing calculator | Signup form | Pass |
|---|
| DeepSeek v4-flashself-check loop | 4/4 | 3/4 | 4/4 | 11/12 |
| DeepSeek v4-flashsingle shot | 3/4 | 4/4 | 4/4 | 11/12 |
| DeepSeek v4-proself-check loop | 4/4 | 4/4 | 4/4 | 12/12 |
| DeepSeek v4-prosingle shot | 3/4 | 3/4 | 4/4 | 10/12 |
DeepSeek v4-flash single-shot passed the todo task 3/4 runs; with a real-browser self-check loop it passed 4/4. Agents self-test and pass; real browsers tell the truth.
Methodology
- Every setup gets the same task prompt, the same spec of human checks, the same temperature (0.7), and the same budget (≤ 2 self-check fix rounds).
- Grading is deterministic: a real headless Chromium clicks, types, reads computed styles and localStorage. No LLM grades the outcome.
- Each cell is run 4× because LLM output is nondeterministic — a single run can flip.
- Tasks and specs live in
arena/tasks/; raw per-run results in arena/results/.
Run your own agent
# add your agent adapter to arena/run.mjs, then:
node arena/run.mjs --agent <your-model>/single --task all --repeat 3
node arena/run.mjs --agent <your-model>/selfcheck --task all --repeat 3
node arena/leaderboard.mjs # regenerate this page
Bring your own model, harness, or framework — the checks are the same for everyone.