What is Voice Standard Bench?
Voice Standard Bench is a Go command-line tool for comparing models available through llm_gateway. It tests summarization, information extraction, text correction, and dictation using AssemblyAI call transcripts or a synthetic dictation dataset. The benchmark records latency and task quality, with results grouped by audio length. Teams can also measure streaming time-to-first-token, review staged results, and build a cumulative result set across multiple runs.
Voice Standard Bench Features:
Four Benchmark Tasks: Run models on summarization, information extraction, text correction, and dictation, selecting tasks with command-line flags.
Bucketed Metrics: Review latency, tokens per second, and quality by audio-length bucket, with latency statistics including minimum, maximum, mean, median, and 95th percentile.
Quality Scoring: Use an LLM judge to score summarization, extraction, and correction outputs; correction is judged against the supplied corrupted transcript.
Mechanical Dictation Scoring: Score dictation against each record's answer key using exact match, word error rate, punctuation, capitalization, and formatting-command accuracy.
Streaming TTFT: Run a separate streaming pass to measure time-to-first-token for the 30-second and 120-second audio buckets, without judging the generated text.
Replayable Runs: Save model outputs and call details as they return, then replay judging with a different judge model without calling the tested models again.
Canonical Results: Merge approved run results into a versioned set, replacing data for combinations tested again while preserving untouched combinations.
Voice Standard Bench Benefits:
Spot Length Effects: Separate results by audio duration to see when a model's speed or quality changes as transcripts get longer.
Compare Tradeoffs: Review quality alongside latency and token throughput to assess model performance across more than one measure.
Avoid Repeated Calls: Reuse saved outputs for new judging, avoiding another set of model calls when changing a rubric or judge.
Keep Coverage Intact: Accumulate results across runs without discarding previously merged data for combinations a run did not test.
Check Before Merging: Inspect a combined review summary before approving results, or enable approval by flag for scripted runs.
Use Cases:
Compare Summaries: Evaluate which available model produces stronger summaries from real call transcripts at different audio lengths.
Test Extraction: Compare model outputs for pulling information from call transcripts when there is no annotated entity list to use as a gold answer.
Assess Corrections: Test models on simulated corrupted transcripts and judge their faithfulness and improvement against the supplied input.
Check Dictation Commands: Measure how accurately models turn spoken commands such as “comma” or “new paragraph” into formatted text.
Measure First Response: Compare streaming responsiveness for short and medium-length audio buckets when the time before the first token matters.