I built a local-first memory for coding agents. It's a Rust binary and one SQLite file, using MiniLM embeddings plus BM25 keyword search, with no LLM in the retrieval path and no re-ranker.
On LongMemEval S (470 scored questions) it scores R@1 0.913 (429), R@5 0.987 (464) and R@10 0.996 (468).
I wanted more, mainly at R@1 and recall_all (every gold session in the top K). But all while preserving the strict rules of NO LLM, no re-ranker model and not tuning for the benchmark.
Claude Code did most of the legwork:
- offline replays of chunk-score dumps
- a DEV/TEST split by question-id hash
- a "guard" suite (my coding eval plus probes against a snapshot of my real palace), so a benchmark gain couldn't quietly hurt real use
What we tried
| Idea | LongMemEval | Real-palace guards |
|---|---|---|
| Fix two WordPiece tokenizer deviations | 0 questions changed | probes -1/200 |
| Embed at MiniLM's trained 128 tokens, not 512 | R@1 +1, recall_all@5 -7 | probes 0.900 -> 0.875, fail |
| 256 tokens | +4/-4 at R@1 | fail |
| Re-weight cosine vs BM25 (0.70 to 0.85), DEV only | +/- 2 questions | probes 0.885 -> 0.790, coding eval 0.922 -> 0.882 |
| Count BM25 only when cosine > 0 | <= 1 question | probes -4 to -6 pts |
| RRF, min-max, z-score, saturating BM25 | all within 3 questions of baseline | n/a |
| Split two-part questions and merge best scores | R@1 +2/-0 DEV, 0/0 TEST | unchanged |
| Treat advice requests as preferences | R@1 -5 | n/a |
The only clean win was 429 -> 431 at R@1. BEAM 100K is already 355/355 at R@5.
How I read it
- One question is 0.2%. Most "improvements" were 1 to 4 questions, moving in both directions.
- The right sessions are already in the top 30 for 469 of 470 questions. The losses come from ordering, not from failing to find them.
- Fusion is saturated: every way of combining cosine and BM25 lands within 3 questions of the others.
- Every LongMemEval gain cost my real-palace probes more than it gained.
- The leftover misses are reasoning, not retrieval: questions about dates and counting across sessions. The paper's own gains there come from an LLM, which I avoid on purpose.
So my answer is both, but mostly the benchmark. For a pure retriever, S is close to its ceiling. At R@1 I'm at the limit of a small embedding model plus BM25. The next real step is probably a local cross-encoder reranker, not more tuning.
The Claude Code part
The most useful thing it did was keep me honest. It flagged when a "win" was 2 questions, reported failed guards plainly, and stopped when I said "stop if we don't have clear wins". Nothing from the week shipped, and I think that's the right result.
Questions
- At 98%+, is LongMemEval S still a useful signal for you, or have you moved to M, BEAM 1M or your own evals?
- Has anyone gotten a real R@1 lift from a local cross-encoder on conversational memory without bad latency?
- Is there a non-LLM trick for the date questions that I'm missing?
Repo, runner and per-question outputs: https://github.com/AncientiCe/palace-rs





