Updated 7 October 2026. This page previously said the 22 April run cleared every production-fit threshold, and that supersedes-aware reranking improved latency without trading accuracy. The corrected reading is below. The numbers in the table were already right. The sentences around them were not. Graph stays off the default retrieval path until real pilot data shows it helps. This benchmark cannot prove that.
Accord Book is built around a core promise: when a team asks for project context, the system returns the decisions, constraints, and history that still matter now.
Benchmarking retrieval is how we hold that promise to account.
Here is what the April 2026 evaluation showed, including what it does not show.
The benchmark run
The 22 April 2026 LongMemEval-style run used a 20-question slice (BENCHMARK_SUBSET_LIMIT=20, first slice). These are the numeric gates we had set. Meeting a gate on 20 questions is not the same thing as a finished production-fit claim. The internal note on this file calls it ship-with-caveats.
| Metric | Baseline | Treatment | Threshold | Result |
|---|---|---|---|---|
| Retrieval p95 | 1431.4 ms | 990.1 ms | < 1500 ms | Pass |
POST-curation llm_correct |
90.0% | 90.0% | >= 70% | Pass |
| Extraction p95 / max | 517.0 s / 536 s | — | < 600 s | Pass |
| Worst curation stage max | 76 s | — | < 180 s | Pass |
| API 4xx/5xx | 0 | 0 | 0 | Pass |
The pipeline did run end to end: ingestion, extraction, curation, retrieval, and an LLM judge. Pre-curation and post-curation scores on this file are the same, including latency down to the reported precision. Curation did not move this slice. Judged correctness is 18 of 20. A 95% Wilson interval on 18/20 is about 70% to 97%, so the lower bound sits on the 70% bar.
String match was 80% (16 of 20). The judge marked two more answers correct than literal overlap did. That gap is why we log both. It is not a reason to drop the string-match number.
What else the same file records
| Metric | Baseline | Treatment |
|---|---|---|
| Retrieval max | 9077 ms | 7926 ms |
| Top result superseded | 0% | 0% |
| Top result has a successor | 0% | 0% |
| Supersedes signals, mean | 0.45 | 0.45 |
| Session-id map healthy | 95% | 95% |
Baseline is V1_SUPERSEDES_BASELINE: supersedes ranking off, active memories only. Treatment is V1_SUPERSEDES_TREATMENT: ranking on, active and superseded memories. The treatment was faster at p95. Accuracy, string match, and session recall did not move. The superseded rate on the top result was 0% on both variants, and the signal average is 0.45 even with ranking switched off. This run does not show that supersession changed which memory won. The latency drop is also not isolated to the reranker, because the memory-status filter changed with it.
One of the 20 rows had an unhealthy session-id map.
What we are not claiming
Graph contribution on every LongMemEval variant since April 2026 was zero. That benchmark is long-context recall. It cannot prove whether a graph helps on a real project. Graph retrieval stays off the default path until a real pilot shows that it does. See Graph Off the Hot Path.
For the architecture argument as we were working it out in May, read Why Production Retrieval Needs More Than Vector Search. For the benchmark method we were using then, read How We Decide When Graph Belongs on the Retrieval Path.