What is evalship?
evalship helps teams test and monitor applications that use AI models and agents. It reviews existing evaluations, checks code changes against them in pull requests, and tracks AI calls in production so teams can investigate issues. It also runs prompt and model experiments against evals to compare changes. There is a free option, with Pro at $29 per month and Business at $99 per month.
evalship Features:
Eval Review: At setup, evalship reviews evaluations for flawed pass or fail logic, uncovered prompt instructions, weak test data, unreliable judges, and other gaps.
Pull Request Checks: On pull requests and CI runs, it reports which evaluations broke, likely causes, and AI calls or prompts that have no tests.
Merge Gating: Teams can block merges when changes break evaluations, keeping regressions from passing into the main branch.
Production Monitoring: It checks production AI calls for issues such as errors, overpromises, unsupported claims, private data exposure, unsafe answers, and frustrated users.
Trace Inspection: Production issues include the underlying calls and agent steps, so teams can inspect what happened or pass the details to a coding agent.
Prompt Experiments: Teams can compare prompt and model changes against their evaluations, see which changes perform better, and get suggestions for further experiments.
Existing Integrations: It supports existing eval frameworks and can receive traces through OpenTelemetry, OpenRouter, or frameworks including LangChain and the Vercel AI SDK.
evalship Benefits:
Earlier Detection: Finding broken or missing tests during development gives teams a chance to address risky AI changes before users encounter them.
More Trustworthy Evals: Reviewing evaluation flaws helps teams avoid green results that hide privacy leaks, incorrect answers, or failures in the grading process.
Faster Investigations: Issue reports linked to calls and agent steps make it easier to pinpoint why a production response went wrong.
Grounded Decisions: Comparing changes against the same evaluations helps teams choose prompts and models based on observed results rather than guesswork.
Less Tool Switching: Support for existing eval frameworks lets teams add checks and monitoring without replacing the evaluation setup they already use.
Use Cases:
Prompt Change Review: Check a pull request that edits a support bot prompt for regressions such as promising refunds beyond the business rules.
Eval Suite Audit: Review a newly created test suite to find cases that pass despite leaking customer data or failing to check an important instruction.
Live Support Monitoring: Track a customer service agent in production for unsupported commitments, incorrect answers, or personal data in its replies.
Agent Trace Debugging: Investigate a production run where an agent's tool calls fail repeatedly or its final response contradicts tool results.
Model Comparison: Test a proposed model or prompt change for a summarization task against existing evals before deciding whether to ship it.

