What is Pith?
Pith is a Python tool for measuring and applying ways to reduce output-token use on large language model routes. It sits between an application and its model provider, records sampled requests, and tests cheaper response profiles against the route’s existing answers. Developers can run it as a standalone proxy or use its integrations with LiteLLM, Headroom, or Portkey.
Pith Features:
Route Tracking: Pith groups requests by provider, model, and system prompt, then records usage and a sample of request bodies in a SQLite file.
Profile Sweeps: After a route has 50 sampled requests, it replays them using profiles that adjust effort or add concision instructions and examples.
Answer Judging: A judge model compares each candidate response with baseline responses and measures the baseline’s own agreement as a quality floor.
Cost-Based Pinning: Pith pins the cheapest profile that meets the floor within sampling error and is projected to repay the sweep within a month.
Holdout Testing: A sweep can reserve a holdout sample to confirm a candidate profile before Pith pins it.
Proxy Integrations: Traffic can pass through Pith’s standalone proxy or its LiteLLM, Headroom, and Portkey integrations, with support varying by integration.
Live Rechecking: The recheck command judges live traffic and can revert a pinned profile if its performance drifts.
Pith Benefits:
Controlled Savings: Teams can reduce token use on routes where testing shows that a cheaper response profile preserves the baseline’s level of answer quality.
Fewer Regressions: Comparing candidate answers with the baseline helps avoid adopting changes that save tokens by dropping information.
Spend Awareness: Factoring sweep costs into the pinning decision helps teams avoid changes that are unlikely to repay their evaluation cost.
Prompt Cache Continuity: Because rewrites leave system prompts untouched, teams can retain prompt-cache hits while applying validated changes.
Safer Failure Handling: When Pith encounters an error, it forwards the original request unchanged, so an optimizer-side problem does not alter the request.
Use Cases:
Support Replies: Evaluate whether a support-ticket response route can use fewer output tokens without losing information customers need.
Ticket Classification: Test a classification route to see whether a lower-effort response profile reduces token use enough to justify its sweep.
JSON Extraction: Check whether an extraction route has unnecessary output tokens to remove while maintaining its existing response quality.
Summary Generation: Assess concision changes for summarisation routes, where shorter answers may omit details present in the baseline.
High-Volume Routes: Review a heavily used model route to determine whether projected monthly savings can repay the cost of testing a profile.

