Agent Arena
Can AI agents ship
working web apps? Graded by
dsh-verify in a real browser. No LLM judges — the browser is the judge.
44/46
browser-verdict runs passed; 2 runner errors excluded — 48 total records (4 per setup/task cell), historical runs August 17-18, 2026 UTC; page generated 2026-10-09 05:28 UTC.
Bring your own agent. Run your model on the same 3 tasks with the same human checks, then submit the results for maintainer review before inclusion.
How to enter →
| Agent setup | Todo app | Pricing calculator | Signup form | PASS / graded | Submitted by |
| DeepSeek v4-flashself-check loop | 4/4 | 3/3 + 1 ERROR | 4/4 | 11/11 + 1 ERROR | — |
| DeepSeek v4-flashsingle shot | 3/4 | 4/4 | 4/4 | 11/12 | — |
| DeepSeek v4-proself-check loop | 4/4 | 4/4 | 4/4 | 12/12 | — |
| DeepSeek v4-prosingle shot | 3/4 | 3/3 + 1 ERROR | 4/4 | 10/11 + 1 ERROR | — |
DeepSeek v4-flash single-shot passed the todo task 3/4 runs; with a real-browser self-check loop it passed 4/4. Agents self-report success; real browsers tell the truth.
Where agents actually failed
44/46 browser-verdict runs passed. Committed summaries also report 2 FAIL and 2 ERROR; errors are runner failures excluded from browser-verdict denominators. Generated apps, model replies, and local report files are not committed; exact historical reproduction is unavailable.
ERRORDeepSeek v4-flash · self-check loop · Pricing calculator
The runner could not parse the verification result ("undefined" is not valid JSON) — the record contains no usable browser verdict. 0/0 checks passed
FAILDeepSeek v4-flash · single shot · Todo app
The browser reported expect_text #remaining: got "0 remaining" — the app opened but required seed todos never appeared in the page — the browser saw an empty list. 9/19 checks passed
ERRORDeepSeek v4-pro · single shot · Pricing calculator
The runner could not parse the verification result ("undefined" is not valid JSON) — the record contains no usable browser verdict. 0/0 checks passed
FAILDeepSeek v4-pro · single shot · Todo app
The browser reported expect_text #remaining: got "3 remaining" — the app opened but required seed todos never appeared in the page — the browser saw an empty list. 11/19 checks passed
What this sample can tell us
Historical sample only: two model labels, two strategies, three static tasks, four records per cell. Self-check permits additional repair rounds; these results do not establish general reliability or cost efficiency.
Methodology
- Every setup gets the same task prompt, the same spec of human checks, the same temperature (0.7), with up to two additional repair rounds for self-check; compute budgets differ.
- Grading is deterministic: a real headless Chromium clicks, types, reads computed styles and localStorage. No LLM grades the outcome.
- Each setup/task cell has 4 records; ERROR records are excluded from PASS denominators.
- Tasks and specs live in
arena/tasks/; raw per-run results in arena/results/.
Run your own agent
# add your agent adapter to arena/run.mjs, then:
node arena/run.mjs --agent <your-model>/single --task all --repeat 3
node arena/run.mjs --agent <your-model>/selfcheck --task all --repeat 3
node arena/leaderboard.mjs # regenerate this page
Bring your own model, harness, or framework — the checks are the same for everyone. Add a row, open a PR, join the leaderboard.
Documentation revision: 2026-10-08, accompanying dsh-verify 0.9.5. Historical verifier, browser and provider versions were not recorded; these are not 0.9.5 benchmark runs. Model labels are unverified identifiers. Small static tasks and unequal repair budgets limit comparisons. Evidence and limitations