TestEvo-Bench

---

TestEvo-Bench is a live benchmark for evaluating AI software engineering agents on realistic test evolution tasks mined from open-source Java repositories.

Unlike benchmarks that isolate tests from production changes, TestEvo-Bench models the real co-evolution of production code and test suites. Every task comes from a real pair of revisions (old → new), and every developer test was validated by executing it on both revisions: a test counts only if its outcome actually changes with the production code.

The benchmark has three tracks, defined by the developer test's outcomes on the old and new code:

• 🟣 Test Update — the old test passes on the old code but not on the new code; adapt it so that it passes on the new code and does not pass on the old code
• 🟠 Test Generation — write a new test for changed or added behavior; it must pass on the new code and not pass on the old code
• 🟢 Test Refinement — the old test still passes on the new code; refine it so that it covers more of the production methods it exercises

Each task is execution-grounded: the agent works in a two-revision repository inside a Docker environment with JDK 8/11/17/21, Maven and frozen offline dependencies, may edit only the target test file, and is graded by replaying its tests on both revisions, with coverage and mutation score against the developer's mutants.

Full and Lite

Lite is a deterministic subset of Full for cheaper evaluation and faster iteration: it removes toy repositories, tasks whose coverage cannot be measured, non-adjacent revision pairs, potentially flaky tests and tasks with more than three target methods, then samples update and generation tasks by year with per-repository caps. Agent results on Lite match those on Full.

Datasets

Each dataset holds one track with two configurations, full (default) and lite:

• Test Update — https://huggingface.co/datasets/TestEvo-Bench/teb-update
• Test Generation — https://huggingface.co/datasets/TestEvo-Bench/teb-generation
• Test Refinement — https://huggingface.co/datasets/TestEvo-Bench/teb-refinement

from datasets import load_dataset

update_lite = load_dataset("TestEvo-Bench/teb-update", "lite", split="test")
generation_full = load_dataset("TestEvo-Bench/teb-generation", "full", split="test")

Links

• 🌐 Website, data explorer and leaderboard — https://www.testevo-bench.com/
• 🤗 Hugging Face Space — https://huggingface.co/spaces/TestEvo-Bench/
• 💻 Evaluation framework (runner, agent and evaluator images) — https://anonymous.4open.science/r/testevo-bench-1150/README.md

---