What is echoChain?
echoChain is a command-line tool for measuring how an answer changes as language models rewrite it repeatedly. It is aimed at developers and people evaluating LLM behavior who want to examine information degradation across a rewrite loop. You can start a run with a prompt or use a packaged seed, then analyze a finished run. An optional baseline extra adds sentence-transformer scores.
echoChain Features:
Rewrite Loop Runs: Run an answer through repeated model rewrites to measure how its content changes across a sequence of transformations.
Custom Starting Text: Start a live run with your own text, such as a factual statement, and specify a maximum number of cycles.
Packaged Seed: Use the included example configuration and seed to run a dry test without starting a live cycle.
Run Analysis: Analyze a completed run by providing its run directory, giving you a separate command for reviewing the results.
Baseline Scores: Install the optional baseline extra to add sentence-transformer scores alongside the tool’s core measurement.
Environment Configuration: Provide an OpenRouter API key through an environment variable or a .env file for live runs.
echoChain Benefits:
Spot Content Drift: Following an answer through multiple rewrites makes it easier to see whether its information is changing over the loop.
Evaluate Stability: Repeated transformations give you a practical way to examine how consistently an answer holds up when rewritten.
Review Completed Runs: A dedicated analysis command lets you inspect a finished run without treating the act of running the test as the whole evaluation.
Test Before Running: The dry-run option lets you try the packaged setup before committing to a live model cycle.
Add a Scoring Baseline: Optional sentence-transformer scores provide another measurement to consider when evaluating changes in an answer.
Use Cases:
Fact Retention Checks: Test whether a technical statement, such as a claim about Redis and I/O, changes after being rewritten repeatedly.
Prompt Evaluation: Run a prompt through a rewrite loop to examine how its resulting answer changes from one cycle to the next.
Model Behavior Review: Use runs to investigate information degradation while assessing how models handle repeated rewriting of an answer.
Evaluation Workflow Setup: Dry-run the packaged seed when checking the example configuration before starting a live evaluation.
Run Result Inspection: Analyze a saved run directory after a test when you need to review what happened during the rewrite sequence.

