Development journal — May 2026. This post documents the benchmark run and our thinking at the time. The corrected account of the same file is Accord Book Retrieval Benchmarks: What the April 2026 Run Showed.
Updated 7 October 2026. The title used to say retrieval was above the production-fit line. The run is 18 of 20 judged correct on 20 questions, against a 70% bar. Retrieval max was 9077 ms baseline and 7926 ms treatment. Pre-curation and post-curation scores match. Top-result superseded rate was 0% on both variants. The internal verdict was ship-with-caveats.
Accord Book is built around a simple promise: when an AI-native project team asks for context, the system should return the decisions, constraints, and history that actually matter. That sounds obvious until you try to do it across weeks of Slack messages, voice notes, commits, docs, and client requests.
We use LongMemEval-style tests as one of the gates for that promise. On this 20-question slice, judged correctness was 90% (18 of 20) and treatment retrieval p95 was under the 1500 ms gate. Those are the numeric gates. They are not a finished production-fit claim. The limits section below is part of the result.
What we measured
The run compares retrieval quality before and after curation on a 20-entry LongMemEval production-fit set. We judge correctness with an LLM-as-judge because exact string matching undercounts correct answers when the system gives the right evidence in different wording.
The architecture is organized around a few stable responsibilities:
- project memory is stored canonically in Postgres with vector search
- graph expansion is a derived layer for richer context, not the source of truth
- retrieved evidence is reranked before it is given to the user or agent
- curation improves the memory layer by turning raw artifacts into structured, provenance-aware knowledge
This separation lets us evaluate retrieval quality, curation quality, and response latency independently.
Latest run
| Metric | Baseline | Treatment | Target | Verdict |
|---|---|---|---|---|
| Retrieval p95 | 1431.4ms | 990.1ms | under 1500ms | Pass |
| Post-curation correctness | 90.0% | 90.0% | at least 70% | Pass |
| Extraction p95 / max | 517s / 536s | n/a | under 600s | Pass |
| Worst curation stage | 76s | n/a | under 180s | Pass |
| API 4xx/5xx | 0 | 0 | 0 | Pass |
p95 on the treatment was under 1500 ms. The max on the same run was 9077 ms baseline and 7926 ms treatment, so the tail is not an interactive budget.
What changed our confidence
String matching reported a lower score than the judge. That is expected in memory retrieval: if the system answers "the OAuth work was descoped because of infrastructure constraints" while the fixture expected a slightly different phrase, a strict string metric can mark a useful answer as wrong.
For Accord Book, the more important question is whether the returned evidence is sufficient for an owner, developer, or coding agent to make the next decision. That is why we treat LLM-judged correctness as the primary public quality signal, with latency and failure rates as guardrails.
Current limits
The latest run is encouraging, but it does not remove the need for continued measurement. The main areas we are watching are:
- tail latency needs continued watching
- lineage density around superseded decisions is still lower than we want
- the benchmark is a gate, not a substitute for pilot usage on real agency data
The practical conclusion is narrower than the original title. This slice met the numeric gates we had written down, and the same file still shows tail latency and low lineage density. The next measurement that matters for graph is not another pass of this benchmark. Graph stays off the default retrieval path until a real pilot shows it helps.