LemonCrow vs. vanilla Claude Code.
"Baseline" = Claude Code with no plugin, no persona, no augmentation. Same model, Docker image, turn cap, and disabled-tools list as the LemonCrow arm.
Anthropic publishes model accuracy on SWE-bench Verified (Opus 4.8: 88.6%) -- no cost, token, or turn-efficiency numbers for Claude Code the CLI against any baseline. LemonCrow publishes that comparison, every raw run committed to the repo below.
| Suite | Baseline correct | LemonCrow correct | Baseline $ | LemonCrow $ | Delta |
|---|---|---|---|---|---|
| SWE-bench Verified 50 tasks × 5 reps, claude-opus-4-8 | 202/250 (80.8%) | 232/250 (92.8%) | $234.84 | $165.45 | +12.0pp correct, 29.5% cheaper, 44.9% fewer tokens, 37.7% fewer turns |
| SWE-bench Lite 10 tasks × 3 reps | 28/30 (93.3%) | 30/30 (100%) | $12.38 | $10.79 | +6.7pp correct, 12.9% cheaper, 20.5% fewer tokens, 15.8% fewer turns |
| SWE-bench Pro 10 tasks × 5 reps -- Go/TS/Python, ScaleAI harness | 44/50 (88%) | 45/50 (90%) | $39.01 | $30.61 | +2.0pp correct, 21.5% cheaper, 35.4% fewer tokens, 28.1% fewer turns |
| Exploration tasks 7 large repos (Tokio, Django, VS Code...) × 5 reps | -- | -- | $19.11 | $6.29 | 67% cheaper, 91% fewer turns, 84% fewer output tokens |
| Telegraphic Q&A 20 engineering Q&A prompts × 5 reps, no code repo | -- | -- | $8.93 | $5.34 | 40.2% cheaper, 46.1% fewer output tokens, input tokens now roughly flat vs. baseline |
| Terminal-Bench 2.1 89 tasks, 1 attempt vs. public tbench.ai leaderboard (5-rep avg) | 78.9% expected | 78.7% | $96.76 | $69.52† | -0.2pp (tie at n=1), 28.1% cheaper† -- floor, not ceiling: 5 LemonCrow runs timed out with real uncounted spend |
* Baseline's turn count includes a hidden session-title API call
(fires without --agent); corrected, baseline ≈1.38 turns
vs. LemonCrow's 1.20 -- cost/output-token deltas unaffected. † 5 of 89
Terminal-Bench LemonCrow runs hit a timeout with real, uncounted spend --
$69.52 is a floor. Full caveats → BENCHMARKS.md.
The true story
LemonCrow wins cost, tokens, and turns on every suite, and correctness on every multi-rep suite. Terminal-Bench (-0.2pp) is single-attempt vs. a 5-rep public average, and costs less anyway.