What is Gauge CLI?
Gauge CLI is an open-source command-line tool for teams that want to make their products easier for coding agents to discover and use. It runs agents against tasks involving a product, collects session evidence, and supports a cycle of finding problems, making changes, and testing again. Local runs use Docker and the credentials or subscriptions for the chosen agent, without requiring a Gauge account or Gauge subscription.
Gauge CLI Features:
Local Agent Runs: Run isolated Docker sessions with Claude Code, Codex, Pi, or OpenCode using your configured credentials.
Task Evaluations: Create saved evaluations with defined tasks and criteria, or use Markdown cases for measurements that can be shared or used in CI.
Product Inputs: Test local Skills, CLIs, SDKs, and files, including web fixtures that let agents encounter unpublished documentation changes.
Session Evidence: Collect normalized agent transcripts, results, and available diffs in the local Gauge workspace for your coding agent to analyze.
Baseline Comparisons: Rerun the same evaluation after a change and compare batches using task completion, available turns or duration, and criterion judgments.
Preference Studies: Create open-ended prompts to examine which product an agent chooses for a task and how its choice changes with configured inputs.
Pre-Release Testing: Install locally built CLI or SDK packages in test sessions so agents can try changes before they are published.
Gauge CLI Benefits:
Find Usage Friction: See where an agent gets blocked, relies on outdated patterns, misses useful guidance, or misunderstands a workflow before those problems reach more users.
Check Fixes Early: Use repeated task results to judge whether a proposed change addresses the original issue and whether it creates new friction.
Keep Work Local: Run measurements without a Gauge account or Gauge subscription, with results and transcripts stored in the local workspace.
Use Existing Agents: Work with supported coding agents and their existing credentials instead of setting up a separate Gauge-hosted evaluation environment.
Ground Decisions: Review session evidence alongside evaluation criteria so product and documentation changes can be based on observed agent behavior.
Use Cases:
Improve CLI Guidance: Test whether a coding agent can complete a multi-command workflow using a CLI's help text and instructions.
Review SDK Changes: Build and test an unpublished SDK package to check whether an agent can use its updated APIs for a defined task.
Test a Skill: Measure whether an agent discovers and follows a Skill when completing a task without being explicitly told to use it.
Revise Documentation: Use a web fixture to expose an agent to draft documentation and check whether the revised setup steps resolve a task failure.
Compare Product Choices: Run a preference prompt to see which tools an agent selects for an open-ended task and whether configured inputs affect that choice.

