← All comparisons
LemonCrow vs Serena
What Serena says, vs. what it scored.
LSP-wrapped semantic code toolkit -- symbol-level navigation and refactoring via real language servers, 40+ languages.
What Serena says about itself
“Serena provides essential semantic code retrieval, editing, refactoring and debugging tools that are akin to an IDE's capabilities, operating at the symbol level and exploiting relational structure.”
“Practically, this means that your agent operates faster, more efficiently and more reliably, especially in larger and more complex codebases.”
What it actually scored — same 14 repos, same 7,213 queries as every other tool
| Tool | MRR | p95 | p100 |
|---|---|---|---|
| ★ LemonCrow +semantic (BGE) | 0.727 | 390ms | 1057ms |
| ★ LemonCrow lexical (default) | 0.676 | 134ms | 319ms |
| Serena | 0.401 | 3834ms | 269001ms |
Real symbol-level LSP tooling, 40+ languages. Cold LSP spin-up shows in latency here (3834ms p95 vs. LemonCrow's 134-390ms). 0.401 MRR.
By query kind -- same benchmark, broken out (no reps in this eval: one deterministic pass per query)
| Kind | LemonCrow +semantic | LemonCrow lexical | Serena |
|---|---|---|---|
| definition | 0.873 (n=1570) | 0.871 (n=1570) | 0.633 (n=1570) |
| content | 0.873 (n=1444) | 0.864 (n=1444) | 0.757 (n=1444) |
| semantic | 0.759 (n=1800) | 0.576 (n=1800) | 0.005 (n=1800) |
| swebench | 0.500 (n=1908) | 0.493 (n=1908) | 0.332 (n=1908) |
| sessions | 0.587 (n=491) | 0.571 (n=491) | 0.339 (n=491) |
n = query/gold pairs of that kind, out of 7,213 total -- every provider scored on all 5 kinds.
By repo -- all 15 repos in the corpus, same query set
| Repo | LemonCrow +semantic | LemonCrow lexical | Serena |
|---|---|---|---|
| astropy/astropy | 0.772 | 0.715 | 0.490 |
| django/django | 0.689 | 0.652 | 0.460 |
| lemoncrow-lab/lemoncrow-dev | 0.467 | 0.477 | 0.316 |
| lemoncrow/lemoncrow | 0.594 | 0.557 | 0.357 |
| matplotlib/matplotlib | 0.801 | 0.747 | 0.000 |
| mwaskom/seaborn | 0.814 | 0.768 | 0.456 |
| pallets/flask | 0.735 | 0.671 | 0.363 |
| psf/requests | 0.840 | 0.803 | 0.482 |
| pydata/xarray | 0.815 | 0.764 | 0.464 |
| pylint-dev/pylint | 0.856 | 0.784 | 0.541 |
| pytest-dev/pytest | 0.826 | 0.739 | 0.545 |
| scikit-learn/scikit-learn | 0.740 | 0.669 | 0.364 |
| sphinx-doc/sphinx | 0.637 | 0.580 | 0.306 |
| sympy/sympy | 0.694 | 0.637 | 0.392 |
| torvalds/linux | 0.726 | 0.668 | 0.428 |
MRR per repo: n-weighted blend across all 5 query kinds, same 7,213-query run.
The true story
Same 14 repositories, same 7,213 query/gold pairs as every tool here, Serena included. Full methodology, every raw number, and the other9 tools →