What is Parsimony?
Parsimony is an open-source benchmark for comparing how much code AI coding models add while solving software tasks. It analyzes published benchmark patches, measuring changes in Python syntax units such as calls, names and literals rather than counting lines. Developers and teams evaluating coding models can view code-size measurements alongside solve results, compare models on shared tasks, or chart results from another benchmark. It measures patch size, not code readability or design quality.
Parsimony Features:
Syntax Unit Counting: Parsimony parses changed Python files and calculates net syntax units by subtracting removed units from added ones.
Published Patch Analysis: It uses patches and pass-or-fail results already published for SWE-bench Verified, without rerunning tests.
Model Leaderboard: The leaderboard ranks models by mean net syntax units added on solved tasks and also shows solve and measurement counts.
Model Comparisons: Charts can compare two metrics or show one metric, and a shared-task view compares models on tasks both solved and measured.
Custom Benchmark Data: Users can fill in an example JSON file with code-size results from another benchmark and open it to create a chart and table.
Chart Export: Charts can be downloaded as PNG images or copied for use elsewhere.
Parsimony Benefits:
See Code Footprint: A syntax-based measure gives model evaluators another way to assess patch size beyond solve rate and cost.
Compare Like Tasks: The shared-task view helps distinguish model comparisons on overlapping tasks from results based on different task sets.
Interpret Results Carefully: Separate solved and measured counts help readers see how much data underlies a model's displayed results.
Reuse Existing Results: Using published patches means evaluating code size does not require rerunning the benchmark's tests.
Bring Your Own Data: The custom-data workflow lets users explore code-size results from a benchmark beyond the ones already shown.
Use Cases:
Select Coding Models: Compare models' measured patch sizes when deciding which coding agent to evaluate or adopt.
Review Benchmark Results: Check syntax changes alongside published SWE-bench Verified solve results when assessing model performance.
Compare Shared Tasks: Use the intersection view to examine two models on tasks both managed to solve and measure.
Explore Model Trends: Filter and chart model results by available categories such as open or proprietary weights and model generations.
Chart Other Benchmarks: Enter code-size results from another benchmark in the example file to view them as a chart and table.