Measured results and limits
All figures below are internal benchmarks run by the Mnemontron team in a development environment. They have not been independently reproduced. Each is reported with its setting; please do not compare them with numbers measured in a different setting.
Retrieval
| Measurement | Setting | Result |
|---|---|---|
| Recall at 20 (BM25) | Natural Questions test, n=3,588, over 21,015,324 Wikipedia passages | 58.5% |
| Recall at 100 (BM25) | Same run | 72.16% |
| Fast-path latency | Same run, median / 95th percentile | 49.3 ms / 105 ms |
| Multi-hop recall at 20 | FRAMES, n=794 | 14.36% |
For reference, Karpukhin et al. (EMNLP 2020, Table 2) report BM25 recall at 20 of 59.1% on a 21M-passage Wikipedia setting. That was a different run, and we cite it only as context, not as a like-for-like comparison.
Answering and abstention
| Measurement | Setting | Result |
|---|---|---|
| Answerable questions answered | Internal held-out set, N=213 (101 answerable) | 90.1% (91 of 101) |
| Unanswerable questions correctly declined | Same set (112 unanswerable) | 88.4% (99 of 112) |
| Query latency | Same set, median / 95th percentile | 40.3 ms / 53.3 ms |
The held-out set was built internally. It is not a customer task set.
Storage and recovery
| Measurement | Setting | Result |
|---|---|---|
| Erasure-coded repair | 10,000 evidence objects, Reed-Solomon 4+2 across 6 buckets, one bucket dropped | 10,000 of 10,000 repaired and verified |
| Disaster-recovery restore | 50,000 evidence objects and claims restored into a fresh stack | Recovery time 101.6 s |
| Ingest with dense vectors | 500,000-item ingest, all items embedded | Completed |
Capacity model
Mnemontron includes a capacity model for scenarios up to 100+ exabytes. It is a projection built from the measurements above and architectural assumptions. It is not a test that was run, and no projected latency or cost should be read as a measured result.
Not measured yet
- Quality on real customer tasks, reviewed by an independent person.
- Production or multi-region performance.
- A web-scale index. Web search has been measured only on free, public data sets; it is not comparable to commercial web search indexes.