HeliosDB Full v8.0: A 14-Protocol Native LSM, RocksDB-Free
v8.0.0 is a milestone release: HeliosDB Full no longer depends on RocksDB. The native LSM tree, custom WAL, content-addressed CAS, and branch-aware GC carry the read and write path end-to-end — while the engine continues to speak 14 wire protocols simultaneously. This post covers what changed, why we did it, and what it means for operators.
Why retire RocksDB
RocksDB is excellent. We used it for years. Three things made the dependency increasingly awkward:
- Branch semantics. Git-like branches with copy-on-write at the storage layer want delta SSTables and branch-aware compaction. We were layering branch metadata on top of RocksDB's column families, which works, but the planner and the GC had to round-trip through a foreign abstraction every operation.
- Time-travel reads. MVCC versions on RocksDB go through
seqnum-based snapshots. That's fine for short-lived snapshots; it's awkward for "give me this row as of timestamp T" with T anywhere in the retention window. - Compile times and binary size. RocksDB pulled in C++ build tooling. Removing it dropped clean-build time materially and shaved ~12 MB off the release binary.
The native LSM
v7.9.0 ported heliosdb-vector off RocksDB first — the easiest module to migrate, and the one with the loudest performance claim. v8.0.0 finished the job: every storage-touching crate is now on the native LSM. Highlights:
- Direct-I/O WAL with single-write durability. Aligned writes via
O_DIRECT; the redundant secondary fsync the prior architecture used was retired in v7.7.3. - Content-addressed value storage. Large values are deduplicated across rows and branches by content hash. Useful for embedding-heavy workloads where the same vector recurs across many tenants.
- Branch-aware GC. Tombstones and stale versions are reclaimed only when no live branch references them; the GC scheduler consults a branch/MVCC matrix before compacting.
- Speculative filters. When a query repeatedly probes a column with a similar value distribution, the engine maintains a probabilistic filter for the next probe.
Three of these four (content-addressed storage, branch-aware GC, speculative filtering) are documented as defensive publications in the HeliosDB IP register. They are public prior art; we describe them here in the same shape.
14 protocols, one storage engine
The wire-protocol adapters — PostgreSQL, MySQL, MongoDB, Redis, Cassandra, Oracle, MSSQL, DB2, Snowflake, Databricks, Pinecone, ClickHouse, HTTP, S3 — all sit above the same query planner sitting above the same storage engine. v8.0 doesn't change that surface; it changes what's underneath.
Performance side-effects
The migration was conservative on perf claims. The numbers we're confident about:
- Branch creation: < 1 ms (was ~10 ms).
- Storage overhead at branch creation: 0 bytes of data, 256 bytes of metadata.
- Read performance on a branch: 97% of main branch.
- Time-travel point queries: same order of magnitude as live reads (was ~3× slower on RocksDB snapshots).
- Cold-build time: ~40% faster (no C++ toolchain).
Operator impact
Storage is on-disk-compatible across the v7.9 → v8.0 cut: the LSM was already shipping in v7.9 with RocksDB sidelined, so the upgrade is zero-touch. The rocksdb Cargo dep is no longer in the dependency tree at all.
What's next
v8.1 (already shipped) hardened the v8.0 implementations for end-user use. v8.2 added the agentic-ops skills repository for Claude Code & Codex. v8.3+ will continue the perf wave: HNSW-on-disk for billion-scale vector search, parallel compaction, async checkpointing.
HeliosDB Full is source-available under SSPL-1.0. Repository: github.com/dimensigon/HDB-HeliosDB-Full.