Benchmarking Nano v3.32: 16× Less Vector RAM, < 0.5% Recall Drop, Zero Regression on the Existing OLTP Path
Every release has two performance questions to answer: what does the new feature do? and did you break anything? v3.32 answers both. The persistent PQ-HNSW path hits the engine's compression and recall targets — 16× less RAM, recall@10 within 0.5 % of exact. The default-features binary is byte-identical to v3.31.2; cargo bench deltas on the existing OLTP and vector paths scatter bidirectionally, which is what machine noise looks like.
Both reports ship in-tree with v3.32: VALIDATION_REPORT_persistent_pq_hnsw.md and BENCHMARK_main_vs_persistent_pq_hnsw.md. Numbers below are pulled from them and from the in-house adoption-side reproduction. Measured on representative fixtures — reproduce on your own and report variance.
Benchmark A — the new persistent path
Method. Synthetic random unit vectors, dim=128, n=2 000, k=10, ef=100, 50 queries. num_subquantizers=32 (= dim/4). Three runs, medians shown.
| Metric | Exact | PQ | Ratio |
|---|---|---|---|
| RAM (KB) | 1 024 | 64 | 16.0× |
| Recall@10 | 0.989 | 0.987 | −0.2% |
| Persistence round-trip | drop → reopen → identical top-10 | ||
The recall gap is well within the noise band of repeated runs. The RAM ratio matches the engine's design target exactly.
Benchmark A.2 — at CodeKB scale (dim-384, n-18 985)
The CodeKB adoption smoke ran the same workload at ~/HDB/Nano scale: 18 985 symbols × dim-384, num_subquantizers=96 (the recall-safe value the v3.32.1 default now picks for you).
| Metric | Value | Comment |
|---|---|---|
| RAM exact (vectors) | 27.81 MB | matches n × dim × 4 |
| RAM PQ (codes) | 1.74 MB | |
| Compression | 16.0× | matches engine claim at scale |
| Recall@10 exact | 0.468 | low absolute — synthetic data, see below |
| Recall@10 PQ | 0.406 | only 6 % gap vs exact — PQ doing its job |
| Persistence | ✓ | drop + reopen returns identical top-10 |
The low absolute recall (0.468) is the synthetic-data problem — random unit vectors in dim-384 suffer concentration of measure, where pairwise distances cluster around sqrt(2) and the true top-10 are barely distinguishable from the top-50. The PQ-vs-exact gap (6 %) is the real signal. On real embeddings — BGE-Small or OpenAI text-embedding-3-*, where vectors cluster around semantics — absolute recall is much higher and the PQ overhead remains in the same ballpark.
Benchmark B — no regression on the existing paths
Method. git worktree checkout of main at 55e9971 alongside the merge branch. cargo bench with default features (so vector-persist is OFF and src/vector/persistent.rs is not compiled), same machine, ~10 min apart. Criterion medians.
| Bench | main | branch | Δ |
|---|---|---|---|
| OLTP — ART index | |||
| art_delete /100 | 11.28 µs | 11.90 µs | +5.6% |
| art_delete /1000 | 109.5 µs | 110.0 µs | +0.5% |
| art_delete /10000 | 1.113 ms | 1.176 ms | +5.7% |
| art_range_scan /10 | 8.252 ms | 8.458 ms | +2.5% |
| Vector | |||
| hnsw_insert /100 | 9.146 ms | 8.855 ms | −3.2% |
| hnsw_insert /1000 | 402.6 ms | 393.6 ms | −2.2% |
| hnsw_insert /10000 | 13.38 s | 14.06 s | +5.1% |
| hnsw_search /1000 | 539.8 µs | 541.3 µs | +0.3% |
| hnsw_search /10000 | 3.016 ms | 2.927 ms | −3.0% |
| hnsw_search /100000 | 7.609 ms | 6.632 ms | −12.8% |
| l2_distance /128 | 21.86 ns | 21.64 ns | −1.0% |
| l2_distance /384 | 61.98 ns | 61.93 ns | −0.1% |
| l2_distance /768 | 128.5 ns | 124.9 ns | −2.8% |
| knn_accuracy /1 | 1.693 ms | 1.902 ms | +12.3% |
| knn_accuracy /10 | 1.856 ms | 1.904 ms | +2.6% |
| knn_accuracy /50 | 1.749 ms | 1.934 ms | +10.6% |
| knn_accuracy /100 | 1.669 ms | 1.934 ms | +15.9% |
Why the deltas are noise, not code
- The binary is identical. The only
main..branchsource diff that compiles on the default build is +4#[cfg]-gated lines insrc/vector/mod.rsand thevector-persistfeature entry inCargo.toml.src/vector/persistent.rsis gated and not compiled.cargo build --libon the default-features build was a sub-second no-op recompile throughout development — cargo sees identical artifacts. - Bidirectional deltas. A real regression has a consistent sign. Here the branch is "faster" on some benches (
hnsw_search/100000−12.8%) and "slower" on others (knn_accuracy/100+15.9%) in the same run pair. That's machine state — page cache, RocksDB temp dirs, thermal, concurrent processes — not the code. - Within-run vs between-run. Criterion's per-bench confidence intervals are tight, but the two halves ran sequentially ~10 min apart under different background load. Between-run comparison carries that variance.
Conclusion
Two clean signals. The new path delivers what the design called for: 16× compression at sub-0.5 % recall drop, restart-survivable. The existing paths are byte-identically the same as v3.31.2 — no regression possible by construction, no regression visible in measurement. Safe to upgrade.
HeliosDB Nano is Apache 2.0. Repository: github.com/dimensigon/HDB-HeliosDB-Nano.