What is eval-skills?
eval-skills is a suite of skills for coding agents that helps teams build and improve evaluations for AI applications. It guides engineers, product managers, domain experts, founders, and teams through inspecting executions, reviewing examples, analyzing failures, validating graders, and testing changes. It works with Claude Code, Codex, Cursor, and other assistants that support Agent Skills. Use it with existing traces and datasets or start by examining a real execution.
eval-skills Features:
Coding Agent Skills: A set of focused skills guides coding agents through evaluation work, from inspecting evidence to maintaining regression checks.
Trace Inspection: The workflow can capture a real execution and make its behavior understandable, with Python and TypeScript tracing templates.
Human Review: Review steps help teams sample realistic cases and record observations about failures and their significance.
Failure Analysis: Annotations can be grouped into failure modes, refined with a reviewer, and prioritized by frequency and impact.
Grader Calibration: Grader tools help check automated judgments against human labels and create failure-specific evaluation checks.
Controlled Experiments: The Descent workflow tests one improvement hypothesis at a time against a baseline, checks regressions, and keeps or reverts changes.
eval-skills Benefits:
Grounded Priorities: Teams can direct engineering effort toward observed problems that matter to users instead of relying on a general quality score.
Clearer Decisions: Human observations and inspected examples give product and engineering teams a shared basis for deciding what good behavior means.
Safer Changes: Comparing application changes against validated checks helps teams spot quality regressions before accepting an improvement.
Reusable Evidence: Existing traces, datasets, evaluators, and review tools can carry forward, avoiding unnecessary repetition of completed evaluation work.
Portable Work: Preserving cases, annotations, and results makes it possible to change how the workflow runs without losing evaluation evidence.
Use Cases:
Prototype Testing: A founder with an AI prototype and no logs can run a request locally, inspect it, and develop reviewed scenarios for likely failures.
Production Debugging: An engineering team can sample production traces, review failures, and turn confirmed examples into regression cases.
Questionable Scores: A team can investigate an existing evaluation by recomputing its headline from raw results and checking grader errors and references.
Grader Validation: A domain expert can compare a proposed judge with expert labels before using its results to assess application behavior.
Change Evaluation: Engineers can test a specific application improvement against an established baseline and check for regressions before keeping it.