Purpose-built storage engine with HTAP dual-format storage, optimized for modern hardware.
Row + columnar in a single engine. OLTP and OLAP on the same data, zero ETL.
HeliosCore is a native storage engine designed specifically for HeliosDB's unique capabilities: database branching, vector search, and time-travel queries. Unlike generic key-value stores that require workarounds for these features, HeliosCore was built from the ground up with them as first-class primitives.
-- Features powered by HeliosCore's
-- purpose-built storage layer
-- Zero-cost branch creation
CREATE BRANCH feature_auth
FROM main AS OF NOW;
-- Native time-travel queries
SELECT * FROM users
AS OF TIMESTAMP '2026-01-15 09:00:00';
-- Vector search with optimized I/O
SELECT content,
embedding <=> $query AS distance
FROM documents
ORDER BY distance
LIMIT 10;
-- All three features leverage the same
-- underlying storage primitivesHeliosCore stores table data simultaneously in both row-oriented and columnar formats, enabling a single database to serve OLTP transactions and OLAP analytics without ETL pipelines, separate infrastructure, or data duplication. The storage format is configurable per column — not per table — giving you fine-grained control that competitors like Oracle In-Memory, TiDB, and SingleStore don't offer at this granularity.
EXPLAIN ANALYZE reports the class and the recommended store.-- Per-column storage mode declaration
CREATE TABLE orders (
id INT PRIMARY KEY,
customer TEXT STORAGE DICTIONARY,
amount FLOAT8 STORAGE COLUMNAR,
created_at INT8 STORAGE COLUMNAR,
status TEXT STORAGE DICTIONARY
);
-- OLTP: point lookups use row store
SELECT * FROM orders
WHERE id = 42;
-- → Point lookup in the row store
-- OLAP: analytics use columnar store
SELECT customer,
SUM(amount),
AVG(amount)
FROM orders
WHERE created_at > 1704067200
GROUP BY customer;
-- → Batch scan of the column store
-- Same table. Same data. No ETL.When data is inserted, HeliosCore writes each value to the appropriate store based on the column's storage mode. Columnar columns are batched into 1,024-row segments for sequential analytics access, while the row store retains lightweight reference markers for point lookups. Multi-store indexes allow the query optimizer to pick the fastest path for every query.
EXPLAIN shows it as “Columnar Scan”.# HTAP Dual-Format Data Flow
INSERT INTO orders (id, amount, status)
↓
+——————————————————————+
| Per-column storage mode |
+——————+———————————————+
↓ ↓
Row Store Column Store
[id, ref, status] [1024-row batches]
Query classification (EXPLAIN):
SELECT * WHERE id=42
→ Workload: OLTP
→ Recommended store: Row Store
SELECT SUM(amount) GROUP BY ...
→ Workload: OLAP
→ Recommended store: Column Store
→ Columnar scan of the batches
Resource budgets (built-in defaults):
OLTP: 60% | 1000 concurrent ops
OLAP: 40% | 16 concurrent ops
# tracked today; enforcement plannedHeliosCore bypasses the operating system's page cache entirely using O_DIRECT, giving the storage engine complete control over what data lives in memory. This eliminates double-buffering overhead and delivers predictable, consistent latency regardless of OS memory pressure.
# Latency comparison under load
# (internal tests, HeliosDB Lite, buffered
# I/O vs O_DIRECT, synthetic workload on an
# 8-core NVMe host; reproduce on your own
# hardware)
With OS Page Cache (buffered I/O):
P50 latency: 0.8ms
P99 latency: 12.4ms # spikes from eviction
P999 latency: 45.2ms # tail latency
With HeliosCore O_DIRECT:
P50 latency: 0.6ms
P99 latency: 1.8ms # predictable
P999 latency: 3.1ms # tight tail
Key insight:
Not just faster median - dramatically
better tail latency. No surprise
stalls from OS memory management.HeliosCore is designed for the hardware of today and tomorrow. It takes full advantage of NVMe SSDs, large memory capacities, and multi-core processors — capabilities that legacy storage engines (designed in the HDD era) cannot fully exploit.
# HeliosCore hardware utilization
NVMe SSD Layer
Queue depth: Multi-queue I/O submission
I/O pattern: Aligned, direct, parallel
Read-ahead: Workload-adaptive prefetch
CPU Layer
Threading: Work-stealing thread pool
SIMD: AVX2/AVX-512 for hot paths
Lock strategy: Lock-free where possible
Memory Layer
Buffer pool: Workload-aware eviction
Allocation: Arena-based, zero-copy
Page sizes: Adaptive (4KB - 2MB)
Result:
Scales linearly with hardware.
More cores = more throughput.
More NVMe = more IOPS.
More RAM = larger working set.HeliosCore manages its own cache with workload-aware eviction instead of relying on the OS page cache. Per-class memory budgets that adapt to your workload are planned.
# Storage cache size (bytes)
[storage]
cache_size = 536870912 # 512 MB default
# Planned:
# - per-class budgets (row / index /
# vector / query memory)
# - access-pattern warm-up on startup
# - a total memory limit with an
# out-of-memory policyHeliosCore treats compression as a first-class feature, not an afterthought. Per-column adaptive compression automatically selects an algorithm for each data type without manual tuning.
# Per-column adaptive compression
# (automatically selected by HeliosCore)
Column: user_id (INTEGER)
Algorithm: Delta + Bit-packing
Column: email (VARCHAR)
Algorithm: FSST (short strings)
Column: created_at (TIMESTAMP)
Algorithm: Delta-of-delta + ZSTD
Column: description (TEXT)
Algorithm: ZSTD (level 3)
Column: embedding (VECTOR(768))
Algorithm: Product Quantization
Resident RAM: ~16x less
# internal tests, Nano 3.32.1 vs an
# unquantized index, recall-safe defaults
Column: status (ENUM)
Algorithm: Dictionary encoding
# Compression ratios depend on your data.HeliosCore's storage layer natively understands database branching. Copy-on-write at the storage level means branch creation is instantaneous regardless of database size, and branches share unchanged data automatically.
-- Create a branch (instant, any DB size)
CREATE BRANCH feature_auth
FROM main AS OF NOW;
-- Switch to branch and make changes
USE BRANCH feature_auth;
ALTER TABLE users
ADD COLUMN role VARCHAR(50);
INSERT INTO users (name, role)
VALUES ('admin', 'superuser');
-- Main branch is completely unaffected
USE BRANCH main;
SELECT * FROM users;
-- No 'role' column here
-- Merge when ready
MERGE BRANCH feature_auth
INTO main;
-- Storage: only modified pages
-- are stored separately.
-- Shared data = zero overhead.HeliosCore provides write-ahead logging with automatic crash recovery. Data durability is guaranteed even during unexpected failures — power loss, kernel panics, or hardware failures. Recovery is automatic and fast.
# WAL durability
[storage]
wal_enabled = true
wal_sync_mode = "sync"
# sync: fsync on every write
# group_commit: batched commits
# async: OS-managed flush
# Recovery is automatic on restart:
# the WAL is replayed from the last
# checkpoint.
# Planned: page checksums and a
# configurable recovery-time target.The Filter Advisor is planned. It will analyse query workloads and choose a probabilistic filter (Bloom, Cuckoo, Xor, Ribbon, MPHF, Count-Min Sketch) for each column, table and segment. The engine does not run the advisor yet: its helios_filter_* system views are present but return no rows until it ships.
-- Filter Advisor views: present today,
-- empty until the advisor ships
SELECT * FROM helios_filter_recommendations;
-- (0 rows)
-- Also present and empty today:
-- helios_filter_metrics
-- helios_filter_budget
-- helios_filter_build_queue
-- helios_filter_typesA staging area for high-throughput ingestion is planned. Incoming data will be written to append-only buffers, sorted into runs and compacted into main storage in the background while the table stays readable. Today, load data with standard SQL over the PostgreSQL wire protocol.
-- Staging statistics view: present today,
-- empty until the staging area ships
SELECT * FROM pg_staging_area_stats;
-- Today: load with standard SQL over the
-- PostgreSQL wire protocol
INSERT INTO orders (id, customer, amount)
VALUES (1, 'acme', 120.50),
(2, 'globex', 75.00);HeliosCore eliminates the compromises that come with building on top of generic key-value stores like RocksDB or LevelDB.
| Characteristic | RocksDB / LevelDB | HeliosCore |
|---|---|---|
| Latency consistency | Spikes during compaction | Predictable P99 |
| Database branching | Not supported | Native, zero-cost CoW |
| Time-travel queries | Snapshot-based (expensive) | Native MVCC versioning |
| Vector data layout | Generic KV (suboptimal) | Vector-aware I/O paths |
| HTAP dual-format | Not supported | Per-column row + columnar |
| Workload routing | Manual (application-level) | Automatic OLTP/OLAP classification |
| Memory management | Block cache + OS page cache | Unified buffer pool (O_DIRECT) |
| Crash recovery | WAL replay + manifest | WAL replay from the last checkpoint |
| Compression | Per-block, single algorithm | Per-column, adaptive |
| Filter/index tuning | Manual (DBA required) | Filter Advisor planned |
| Bulk ingestion | Generic write path | Staging area planned |
Per-column row + columnar storage with automatic workload routing. OLTP and OLAP on the same data, zero ETL, with automatic OLTP/OLAP query classification.
O_DIRECT I/O and the engine's own cache keep tail latency independent of OS page-cache evictions.
Copy-on-write at the storage level enables zero-cost branch creation. No external tool or workaround needed.
Per-column algorithm selection with no manual tuning. About 16× less resident vector RAM at recall-safe defaults (internal tests, Nano 3.32.1 vs an unquantized index, see benchmarks.html).
Bypasses OS page cache for predictable memory usage. No double-buffering, no surprise evictions.
WAL replay on restart, proportional to the changes since the last checkpoint rather than to database size.
Per-column choice of probabilistic filters from workload statistics. Its helios_filter_* views are present but not yet populated.
High-throughput ingestion through append-only staging buffers with background compaction.
HeliosCore powers the storage layer in both HeliosDB Lite and HeliosDB Full editions. HTAP dual-format storage, Direct I/O and adaptive compression — start building with the next-generation storage engine today.