What is Carmen?
Carmen is a command-line tool for checking whether AI-written Metal GPU kernels are correct and whether they improve model performance on Apple Silicon Macs. It is aimed at developers experimenting with local language-model inference, with a terminal interface for measuring models and kernels rather than relying only on kernel-level benchmarks. Claude can generate kernels, while measuring bundled options does not require an API key.
Carmen Features:
Metal Kernel Generation: Claude writes Metal GPU kernels for specified operations, and Carmen tests and iterates on the candidates.
Correctness Testing: Tests cover hundreds of inputs, including hidden cases Claude does not see, and report where and why a kernel fails.
Bug-Test Validation: A bug-checking command compares Carmen's tests with a usual one-input check against planted kernel bugs.
Performance Comparisons: The speedup command compares stock MLX, mx.compile, and Carmen's bundled kernels on selected models.
Model-Level Measurement: Carmen checks whether kernel results affect real-model performance on the Mac, not just isolated operation timings.
Inference Levers: The levers command measures speculative decoding, KV-cache quantization, and 3-bit weights against stock MLX.
Carmen Benefits:
Fewer False Wins: Hidden inputs and bug-focused tests make it harder to mistake a kernel that passes one example for a reliable implementation.
More Useful Benchmarks: Real-model checks show whether a faster operation translates into faster inference on the hardware being tested.
Clearer Comparisons: Side-by-side measurements help developers see how custom kernels compare with MLX and its automatic fusion.
Local Evaluation: Bundled model and inference measurements can be run on an Apple Silicon Mac without an API key.
Better Optimization Focus: Measuring inference techniques alongside kernels can reveal when memory limits, rather than kernel speed, constrain a model.
Use Cases:
Kernel Experiments: Developers can ask Claude to write a Metal kernel for an operation and test candidate versions on their Mac.
Kernel Verification: GPU-kernel developers can check whether implementations handle varied and hidden inputs before treating them as correct.
Model Benchmarking: Local inference developers can compare stock MLX, automatic fusion, and bundled kernels across available models.
Decode Optimization: A developer can measure whether speculative decoding changes generation speed for a model and prompt length.
Quantization Checks: Teams testing lower-precision KV caches or 3-bit weights can measure speed and inspect output drift against stock MLX.

