What is agentfork?
agentfork is a local testing and debugging tool for people building LLM agents in Python. It records an agent’s model and tool calls, then lets you restart from a chosen step with a prompt, model, or tool-result change. You can run several versions repeatedly and compare their results, rather than rerunning the whole agent for every experiment. It works through a command-line interface or a local web UI, and supports OpenAI, Anthropic, LangChain, and LangGraph workflows.
agentfork Features:
Run Tracing: Decorators and client wrappers record agent runs, including each model and tool call, its inputs and outputs, tokens, and latency.
Step Forking: Fork a recorded run at a selected step, change its model, system prompt, or tool result, and replay earlier steps from cache.
Variant Sweeps: Run multiple changes several times, in parallel, and compare pass rates, tokens, elapsed time, and estimated cost.
Custom Checks: Check results against expected text or a regular expression, or use a Python function that can inspect the final output and recorded steps.
Run Comparison: View run timelines and step details, compare runs side by side, and identify the first step where their recorded behavior differs.
Replay Tests: Save a recorded run as a pytest test that replays from cached data and reports differences in inputs, steps, or final output.
agentfork Benefits:
Less Repeated Work: Replaying unchanged steps from cache avoids paying to run those parts of an agent again during focused experiments.
Evidence-Based Fixes: Repeated trials and pass rates make it easier to distinguish a dependable change from a result that worked once by chance.
Clearer Debugging: Step-level inputs and outputs help pinpoint where an agent went wrong instead of relying only on its final response.
Cost Awareness: Per-variant token, time, and cost comparisons help assess trade-offs before adopting a more expensive model or prompt change.
Local Data Control: Runs stay on the user's machine, with no account or remote server required to browse or test them.
Use Cases:
Prompt Evaluation: Test alternate system prompts at the step where an agent misinterprets instructions, then check which version passes repeatedly.
Model Selection: Compare models on the same recorded agent step to see whether a model change improves the result enough to justify its cost.
Tool Result Testing: Change a tool's recorded result to investigate how different API responses affect the agent's later decisions.
Agent Regression Checks: Turn a known run into a replayable pytest test and use it to detect changes to recorded behavior in CI.
Multi-Input Validation: Sweep a prompt or model change across several recorded runs to check that a fix holds for different user requests.

