LemonCrow vs. vanilla Claude Code.
"Baseline" = Claude Code with no plugin, no persona, no augmentation. Same model, Docker image, turn cap, and disabled-tools list as the LemonCrow arm.
Anthropic publishes model accuracy on SWE-bench Verified (Opus 4.8: 88.6%) -- no cost, token, or turn-efficiency numbers for Claude Code the CLI against any baseline. LemonCrow publishes that comparison, every raw run committed to the repo below.
| Suite | Baseline correct | LemonCrow correct | Baseline $ | LemonCrow $ | Delta |
|---|---|---|---|---|---|
| SWE-bench Verified 50 tasks × 5 reps, claude-opus-4-8 | 202/250 (80.8%) | 232/250 (92.8%) | $234.84 | $165.45 | +12.0pp correct, 29.5% cheaper, 44.9% fewer tokens, 37.7% fewer turns |
| SWE-bench Lite 10 tasks × 3 reps | 28/30 (93.3%) | 30/30 (100%) | $12.38 | $10.79 | +6.7pp correct, 12.9% cheaper, 20.5% fewer tokens, 15.8% fewer turns |
| SWE-bench Pro 10 tasks × 5 reps -- Go/TS/Python, ScaleAI harness | 44/50 (88%) | 45/50 (90%) | $39.01 | $30.61 | +2.0pp correct, 21.5% cheaper, 35.4% fewer tokens, 28.1% fewer turns |
| Exploration tasks 7 large repos (Tokio, Django, VS Code...) × 5 reps | -- | -- | $19.11 | $6.29 | 67% cheaper, 91% fewer turns, 84% fewer output tokens |
| Telegraphic Q&A 20 engineering Q&A prompts × 5 reps, no code repo | -- | -- | $8.40 | $4.48 | 46.7% cheaper, 60.9% fewer output tokens, 34.5% faster (every token category fell this run -- cost and tokens agree) |
| Terminal-Bench 2.1 89 tasks x 5 reps vs. Claude Code 2.1.205 leaderboard (5 reps, matched) | 351/445 (78.9%) | 351/445 (78.9%) | $73.75 | $61.98 | Tied on correctness (351/445 both sides); 98.6% fewer fresh input tokens (182K vs 12.87M) and 33.8% fewer output; 16.0% cheaper once cache-write pricing is normalized to a matched tier |
* Baseline's turn count includes a hidden session-title API call
(fires without --agent); corrected, baseline ≈1.42 turns
vs. LemonCrow's 1.43 -- essentially tied once that title-generation
round trip is stripped out; cost/output-token deltas unaffected. Full caveats → BENCHMARKS.md.
The true story
LemonCrow wins tokens and turns on every suite, and correctness on most multi-rep suites. On Terminal-Bench, matched at 5 reps on both sides, LemonCrow ties baseline on correctness (78.9% vs 78.9%) while sending 98.6% fewer fresh input tokens and costing 16.0% less once cache-write pricing is normalized to a matched tier.