Evaluation & training data for coding agents
Pushback AI builds and curates SWE-Bench and Terminal-Bench task data for teams training and evaluating coding and agentic systems. We optimise for signal rather than volume — problems that demand real reasoning, multi-step execution, debugging and live environment interaction.
Difficulty profile we target
8 attempts / taskIllustrative. We calibrate acceptance so a strong frontier model clears a task only occasionally across repeated attempts. Anything that saturates — solved cleanly every run — gets cut before it reaches a customer.
01 What we build
Each collection follows the public harness format, so tasks drop straight into an existing evaluation pipeline with no adapter work. Difficulty distribution and task composition can be tailored to what you are measuring.
SWE-Bench format
Real problems in real codebases: bug reproduction and repair, feature work across several files, dependency and build breakage, performance regressions, failing-CI triage. Solving one requires reading unfamiliar code, forming a hypothesis and defending it against a test suite.
Terminal-Bench format
Tasks that can only be finished by operating an environment: standing up a toolchain, recovering broken state, chasing a failure through logs and processes, wiring services together, driving long multi-step sequences without losing the thread.
02 Coverage
Breadth matters as much as difficulty — a set concentrated in one language or one kind of bug stops separating models quickly. Ours is distributed deliberately across all three axes.
Highlighted languages carry the largest share of the current collections. Composition is not fixed — if your evaluation needs weight in a particular language, domain or task shape, we build the slice to your distribution.
03 Verification
Every task runs the same pipeline, in order. A task that fails any stage is repaired or discarded — it does not ship with a caveat.
Problems are drawn from permissively licensed work with traceable provenance. Licence and origin travel with the task.
Each task builds into a deterministic container at a fixed digest and runs offline at evaluation time, so a result is the model's, not the network's.
Fail-to-pass and pass-to-pass sets are separated, so a solution has to fix the stated problem without quietly breaking the rest of the repository.
A reviewer strips hints from the problem statement and checks the fix is not sitting in the visible history, comments or adjacent branches.
Every task is run repeatedly against frontier models. Tasks solved consistently are dropped; tasks solved never are re-examined for solvability before they stay.
Repeat runs catch nondeterministic tests, ordering and timing dependencies, and checkers that pass for the wrong reason.
04 What you receive
Delivered as JSONL plus environment definitions, in the layout the public harnesses already expect.
Samples
We keep a standing sample set from both collections — a healthy number of tasks across several languages and domains, weighted toward the hard end. It's enough to judge difficulty, structure, harness fit and review quality for yourself. Tell us what you're evaluating or training and we'll pick the slice that matches, then scope a larger set around your difficulty distribution and task composition.