What is agent-trials?
agent-trials is a command-line tool for developers who want to compare coding agents on work drawn from their own repositories. You define tasks and shell-based checks in a JSON suite, then run that suite through installed agent CLIs using temporary working copies. It reports task outcomes, elapsed times and available cost data in a scoreboard, with optional JSON results and a Markdown report. It is intended for practical, repeatable checks, not as a general model benchmark or security sandbox.
agent-trials Features:
Custom Task Suites: Define task prompts, verification commands, optional setup steps and timeouts in a trials.json file that can live alongside the code it describes.
Agent CLI Selection: Run tasks with built-in Claude, Codex and Gemini specifications, or define custom command arguments that receive each task prompt.
Temporary Task Copies: Each agent-task run gets a temporary copy of the repository, excluding node_modules, .git, secret-looking files and symlinks by default.
Checker Integrity: Restore configured checker paths from the original repository after an agent runs, and fingerprint named checker files to detect changes.
Result Reporting: View pass or fail results, run times, medians and available cost data, then write a Markdown report or raw per-cell JSON results.
Execution Safeguards: Drop most environment variables, gate known approval-bypass flags behind --unsafe and stop timed-out process groups, while leaving agents outside a true security sandbox.
agent-trials Benefits:
Repository Fit: Results come from tasks in your own codebase and checks, making agent comparisons more relevant than scores on unrelated public benchmarks.
Less Repeated Setup: A saved task suite can be run against multiple agents, avoiding the need to recreate the same evaluation for each candidate.
Cost Awareness: When an agent includes costs in its JSON output, reported spending helps you weigh task results against the quota used.
More Reliable Checks: Protected checker files and explicit verification reduce the chance of treating a successful exit or a rewritten test as a correct fix.
Comparable Runs: Keeping tasks with the repository makes it practical to rerun evaluations after changing agents or model versions and review the results.
Use Cases:
Pre-Release Bug Fixes: Prepare representative bugs from a codebase, ask candidate agents to fix them and check whether the required tests pass before shipping.
Model Version Changes: Rerun the same task suite after switching model flags or agent CLI versions to see how task outcomes and run times differ.
Agent CLI Selection: Compare installed coding-agent CLIs on identical task definitions before choosing which one to use for work in a particular repository.
Repository Smoke Tests: Maintain a small collection of code tasks and shell checks as a reusable smoke-test pack for people and coding agents.
Routine Fix Evaluation: Review whether candidate agents can handle recurring fixes, using verification results and available cost details to inform the choice.