What is GEAR?
GEAR is an open-source framework for evaluating and refining an AI agent’s skills, tools, and workflows against real-world tasks. It brings recursive self-improvement to an agent’s harness, using task feedback to guide changes to how the agent works. Teams can use it with an agent they already have, then inspect results from real tasks to see whether an optimized harness performs better than a general-purpose setup.
GEAR Features:
Task Evaluation: GEAR evaluates agent skills, tools, and workflows against real-world tasks, giving teams a way to assess how an agent’s harness performs.
Feedback Refinement: It uses task feedback in a recursive self-improvement process to refine the harness over repeated evaluation cycles.
Failed Run Review: Its optimization workflow starts by inspecting failed runs to identify where an agent did not complete task requirements.
Candidate Testing: GEAR can propose harness changes and evaluate candidate versions against tasks before deciding which versions perform better.
Existing Agents: The framework is designed to tune the harness around an agent that a team already uses rather than requiring a new general-purpose agent.
Benchmark Results: Its experiments report task pass rates and objective completion results for tasks including Marketing public tasks and Terminal-Bench 2.1.
GEAR Benefits:
More Completed Tasks: In a reported Marketing experiment using 100 public tasks, task pass rate rose from 27% to 53% for Luna with an optimized DSH harness compared with its original setup.
Benchmark Improvement: On Terminal-Bench 2.1, reported Luna task pass rate rose from 52.87% to 84.26%, though the comparison changed both the harness and reasoning effort.
Cost Comparison: For Marketing tasks on the same optimized DSH harness, reported API-equivalent average cost was $0.0568 per task for Luna max and $1.1858 for Astra max.
Evidence for Changes: Task-based evaluation helps teams judge harness changes by observed performance rather than relying only on assumptions about what an agent can do.
Keep Useful Versions: Retaining versions that perform better helps teams avoid losing harness changes that have already improved results on their chosen tasks.
Use Cases:
Marketing Task Testing: Evaluate an agent on public marketing tasks and compare its task pass rate before and after tuning its harness.
Terminal Benchmarking: Test harness changes against Terminal-Bench 2.1 when measuring an agent’s performance on that benchmark.
AutomationBench Review: Work through an AutomationBench optimization by reviewing failed runs, suggesting harness changes, testing candidates, and keeping better-performing versions.
Harness Diagnosis: Inspect unsuccessful task runs to find where an agent’s skills, tools, or workflows may need refinement.
Agent Tuning: Improve the setup around an existing agent when a general-purpose harness misses requirements on the tasks that matter to a team.