HeliosDB-Lite v2.2.0 Release Notes

Note (current distribution): HeliosDB Lite is distributed to licensed customers as a source distribution that builds the heliosdb-lite binary. Upgrade steps on this page use that binary, SQL and the configuration file; see Installation and the CLI reference.

Codename: “Quality & Performance”

Highlights

HeliosDB-Lite v2.2.0 adds full ACID compliance, query optimization and production monitoring.

Key Achievements:

Full ACID Compliance: Snapshot isolation with MVCC guarantees data consistency
Crash Recovery: Complete WAL replay ensures zero data loss after crashes
Intelligent Query Planning: Real statistics drive 10-100x performance gains for complex queries
Production Monitoring: Comprehensive system views for observability
Enterprise Compression: FSST string compression with persistent dictionaries

New Features

Storage & Durability

MVCC Snapshot Isolation

Complete multi-version concurrency control implementation providing true snapshot isolation:

Read-your-own-writes semantics maintained
Transactions see consistent data snapshots
Version resolution via reverse timestamp indexing (O(log N))
Automatic fallback for non-versioned keys (metadata, catalog)

Benefits:

Prevents dirty reads, non-repeatable reads, and phantom reads
Enables long-running analytical queries without blocking writes
Full ACID guarantee compliance

Performance: <2x overhead vs non-MVCC reads

WAL Replay for Crash Recovery

Write-ahead log replay with two-pass algorithm:

Pass 1: Transaction state tracking

Process Begin/Commit/Abort operations
Build transaction state maps
Identify incomplete transactions

Pass 2: Operation application

Skip operations from aborted transactions
Apply committed operations to restore state
Resilient error handling (continues on non-fatal errors)

Supported Operations: Insert, Update, Delete, CreateTable, DropTable, Begin/Commit/Abort

Features:

Transaction-aware replay handles partial transactions
Idempotent (safe to replay multiple times)
Progress reporting (logs every 1000 operations)
Comprehensive debug logging

Impact: Data durability guaranteed after crashes

Table Name Extraction

Robust key format parser for WAL context.

Benefits:

Point-in-time recovery with table context
Improved debugging (crash logs show affected tables)
Table-specific recovery from WAL
Better observability

Query Optimization

Real Statistics Collection

Statistics collection and analysis system:

Table Statistics:

Row count tracking
Average row size calculation
Total table size estimation
Last analyzed timestamp

Column Statistics:

Distinct value count (cardinality)
NULL fraction
Min/max value ranges
Average column width

Selectivity Estimation: Selectivity is estimated for equality, inequality, range, AND, OR, and IS NULL predicates.

Integration: Replaces hardcoded estimates with real data

Impact:

Better query plan quality
Optimal join ordering
Accurate row count estimates

Cost-Based Join Selection (v2.1.0, Enhanced)

Intelligent join algorithm selection based on actual costs:

Algorithms Supported:

Hash Join: O(left + right) for large tables with equality predicates
Nested Loop Join: O(left × right) optimal for small tables or non-equality joins

Decision Logic:

With cost estimator: Choose algorithm with lower cost
Without cost estimator: Heuristic fallback (equality → hash, non-equality → nested loop)

Example Scenarios:

Small tables (100 × 100): Hash join preferred
Large tables (100K × 100K): Hash join critical (3333x speedup)
Non-equality predicates: Nested loop required

Overhead: <2μs per join planning decision

Protocol & Compatibility

PostgreSQL Schema Derivation

Automatic schema extraction for prepared statements:

Implementation:

Parse SQL query to AST using sqlparser-rs
Create logical plan using query planner with catalog context
Extract schema from logical plan
Store schema in the prepared statement
Return schema in Describe messages (RowDescription)

Type Mapping:

Automatic HeliosDB → PostgreSQL OID conversion
Supports all standard types (INT, TEXT, FLOAT, JSON, TIMESTAMP, etc.)

Performance: ~160μs overhead during Parse phase (one-time cost)

Benefits:

Full extended query protocol support
Works with all PostgreSQL client libraries (psycopg2, node-postgres, etc.)
Clients can determine result column types without executing queries
Improved client compatibility

Monitoring & Observability

Transaction Statistics Tracking

Thread-safe atomic counters for database activity:

Metrics Tracked:

xact_commit: Committed transaction count
xact_rollback: Rolled back transaction count
Tuple operations: inserts, updates, deletes
Real-time statistics capture

Features:

Atomic counters (lock-free performance)
Point-in-time snapshot capture
Integration with pg_stat_database system view

Enhanced System Views

Branch Parent Name Lookup:

pg_branches view displays actual parent names (not just IDs)

Snapshot Size Calculation:

Enables snapshot capacity planning

Compression Value Tracking:

Added value count to compression statistics
FSST encoder tracks count as number of strings compressed
Per-value compression efficiency analysis

Vector Index Length:

Accurate vector count from underlying HNSW indexes
Proper monitoring and query planning statistics

Query Statistics Integration:

pg_stat_optimizer returns real statistics
EXPLAIN plans show actual cardinality estimates

Compression Enhancements

FSST String Compression

Fast Static Symbol Table (FSST) compression:

Dictionary Persistence:

Custom binary format serialization
Exports 255 symbols (9 bytes each: 8-byte data + 1-byte length)
Full codec restoration from stored dictionaries
Roundtrip validation ensures compression consistency

Optimized Batch Processing:

Pre-allocated batch compression: 10-15% faster
Adaptive batch compression: evaluates the compression ratio and skips incompressible data
64-item chunked processing for better cache locality

Intelligent Sampling Strategies:

Stratified: Best for uniform data
BeginningWeighted: Good for time-series
Diversity: Best for high-cardinality data
Auto: Selects a strategy based on the data

Performance:

2.0-2.5x compression ratio on emails
1.8-2.2x compression ratio on URLs
<5% CPU overhead
Dictionary training: <200μs for 10K items

Compression Configuration API

Fluent builder pattern for compression configuration:

Features:

Per-column compression type selection (FSST, ALP, None)
Configurable block sizes and dictionary limits
Statistics reporting (compression ratios, space saved)
System view integration (pg_compression_stats)

Performance Improvements

Baseline vs v2.2.0

Indicative figures; reproduce on your own hardware.

Operation	v2.1.0 Baseline	v2.2.0 Target	Improvement
SELECT (in-memory)	50μs	35μs (est)	30% faster
INSERT (with WAL)	100μs	100μs	Maintained
Transaction Commit	200μs	140μs (est)	30% faster
Query Planning	Variable	15% faster	Better statistics
Compression Throughput	Baseline	20% faster	SIMD + batching

MVCC & Time-Travel Performance

AS OF query overhead: <2x vs current-time queries
Snapshot lookup: O(1) in-memory metadata access
Version resolution: O(log N) using reverse timestamp indexing
Snapshot GC: >1000 keys/sec throughput

Breaking Changes

None - This release maintains full backward compatibility with v2.1.x.

API Compatibility

All v2.1.x SQL, PostgreSQL wire clients and configuration files continue to work without modification. In-process code written against v2.1.x also works unchanged (embedded API, licensed source distribution).

Configuration Changes

None required - New features are enabled by default with zero configuration:

MVCC snapshot isolation: Automatic
WAL replay: Automatic on database open
Statistics collection: no SQL step; HeliosDB Lite has no standalone ANALYZE statement

Known Limitations

Deferred to v2.3.0

Sync Protocol Implementation:

Change log query support
Conflict detection and resolution
HTTP/gRPC communication layer
Delta application logic
Client-side conflict handlers

Incremental Materialized Views:

Change tracking for base tables
Delta computation logic
CPU-aware refresh scheduling

Remaining Work

Background Worker:

Auto-refresh for materialized views (currently manual)
CPU monitoring and throttling
Staleness threshold detection

System View Completion

Partial Implementation:

Some system views return placeholder data
Full implementation targeted for v2.3.0

Upgrade Guide

From v2.1.x to v2.2.0

Step 1: Backup Your Database

# Stop your application
# Copy database directory
cp -r ./mydb.helio ./mydb.helio.backup

Step 2: Build the New Binary

Build v2.2.0 from the licensed source distribution and keep the v2.1.x binary for a rollback:

cd heliosdb-lite
cargo build --release
export PATH="$PWD/target/release:$PATH"

Step 3: No Configuration Changes Required

v2.2.0 is a drop-in replacement. All new features are either automatic or opt-in:

Automatic Features:

MVCC snapshot isolation (zero configuration)
WAL replay on database open (automatic crash recovery)
Transaction statistics tracking (automatic)

Opt-In Features:

-- Query statistics
SELECT * FROM pg_stat_optimizer;

-- View compression stats
SELECT * FROM pg_compression_stats;

-- Transaction statistics
SELECT * FROM pg_stat_database;

Step 4: Verify Migration

Open the database with the new binary. WAL replay runs automatically when the server or the REPL opens the data directory, so a directory left by an unclean shutdown is recovered here:

heliosdb-lite repl --data-dir ./mydb.helio

-- Verify data integrity
SELECT COUNT(*) FROM users;

-- Transactions now run under snapshot isolation
BEGIN;
SELECT * FROM users;
COMMIT;

Step 5: Optimize Performance

-- Query optimizer statistics
SELECT * FROM pg_stat_optimizer;

-- Test query performance
EXPLAIN SELECT * FROM users JOIN orders ON users.id = orders.user_id;
-- Shows the chosen plan, including the join algorithm

Migration Checklist

Backup database directory
Build the v2.2.0 binary (cargo build --release in the source distribution)
Restart the server or application (WAL replay automatic)
Verify data integrity with COUNT queries
Test transaction isolation (MVCC active by default)
Review system views for monitoring

Rolling Back (If Needed)

# Stop the server or application
# Restore backup
rm -rf ./mydb.helio
cp -r ./mydb.helio.backup ./mydb.helio

# Start the v2.1.x binary you kept on the restored directory

Compatibility

PostgreSQL Protocol

Compatibility Level: 95%+

Supported Features:

Extended Query Protocol (Parse, Bind, Execute, Describe)
Simple Query Protocol
Prepared statements with schema derivation
Authentication (MD5, SCRAM-SHA-256)
SSL/TLS connections
COPY protocol

Limitations:

Some advanced prepared statement features not yet supported
Sync protocol (deferred to v2.3.0)

SQL Standard

Compliance: SQL-92 + PostgreSQL extensions

Supported:

SELECT, INSERT, UPDATE, DELETE
JOINs (INNER, LEFT, RIGHT, FULL)
Subqueries and CTEs (WITH)
Aggregations (GROUP BY, HAVING)
Window functions
MVCC snapshot isolation (AS OF)

Rust Version

Minimum Required: Rust 1.70+

Recommended: Rust 1.75+ (latest stable)

MSRV Policy: 6-month trailing support

Platforms

Tier 1 (Fully Tested):

Linux (x86_64, ARM64)
macOS (Intel, Apple Silicon)

Tier 2 (Community Supported):

Windows (x86_64)
FreeBSD (x86_64)

Architecture Support:

x86_64 (full SIMD support)
ARM64 (full NEON support)
WASM (experimental, limited features)

Dependencies

Core Dependencies:

RocksDB 0.22+ (storage backend)
Arrow 52+ (columnar format)
Tokio 1.40+ (async runtime)
sqlparser 0.53+ (SQL parsing)

Optional Dependencies:

hnsw_rs 0.3+ (vector search)
aes-gcm 0.10+ (encryption)

Summary

HeliosDB-Lite v2.2.0 adds:

Full ACID compliance (MVCC + WAL)
Query optimization
Crash recovery
Monitoring
Better protocol compliance

Support & Resources

Community

Product page: https://www.heliosdb.com/lite.html
Issues: Report bugs and feature requests to support@heliosdb.com
Community: Discord

Professional Support

Contact the HeliosDB team for enterprise support options
Custom feature development available
Performance tuning and optimization consulting

Version: 2.2.0 Codename: “Quality & Performance”