Changelog

All notable changes to HeliosDB Lite will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[3.1.0] - 2025-12-08 - Multi-User ACID In-Memory Mode

Highlights

HeliosDB-Lite v3.1.0 introduces comprehensive multi-user ACID transaction support for in-memory mode with user-triggered persistence:

• Multi-User Sessions: Concurrent user sessions with independent transaction contexts
• Advanced Locking: Deadlock detection, configurable timeouts, three lock modes
• Isolation Levels: READ COMMITTED, REPEATABLE READ, SERIALIZABLE
• Dump/Restore: Manual and automated backup/restore with compression
• Resource Quotas: Per-user memory limits, connection limits, timeouts
• 100% Backward Compatible: All v3.0.0 code works without changes

Added

Session Management

• Session Manager: Multi-user session coordination with per-user contexts
• Session Lifecycle: Create, authenticate, timeout, and cleanup sessions
• Session Statistics: Track transactions, queries, memory usage per session
• User Authentication: Trust, password, and SCRAM-SHA-256 auth methods
• Resource Quotas: Per-user memory limits, connection limits, transaction timeouts
• Session Monitoring: pg_stat_activity system view for active sessions

Lock Management

• Lock Manager: Advanced lock acquisition with deadlock detection
• Lock Modes: Shared (S), Exclusive (X), Update (U) locks
• Deadlock Detection: Wait-for graph based cycle detection
• Lock Timeout: Configurable timeout with exponential backoff
• Lock Statistics: Track lock acquisition time, contention, deadlocks

Transaction Isolation Levels

• READ COMMITTED: Fresh snapshot per statement, no dirty reads
• REPEATABLE READ: Consistent snapshot throughout transaction
• SERIALIZABLE: Full serializability with conflict detection
• Per-Session Configuration: Set isolation level per session or transaction
• SQL Support: BEGIN TRANSACTION ISOLATION LEVEL... syntax

Dump/Restore System

• Dump Manager: Export/import database to portable dump files
• Full Dumps: Complete database snapshot with schema and data
• Incremental Dumps: Append-mode dumps with LSN tracking
• Compression: zstd (default), lz4, or no compression
• Selective Dumps: Dump specific tables only
• Dump History: Track dump operations with pg_stat_dump_history view
• Dirty State Tracking: Track uncommitted changes, warn on shutdown
• Automated Scheduling: Cron-style scheduled dumps via configuration

CLI Commands

• heliosdb-lite dump: Dump in-memory database to file
  • Options: --output, --compress, --tables, --append, --verbose
• heliosdb-lite restore: Restore from dump file
  • Options: --input, --mode, --tables, --on-conflict, --verbose
• heliosdb-lite status: Show database status and statistics
  • Options: --format (text/json)
• Enhanced start command: Multi-user session support
• Enhanced repl command: Session-aware interactive shell

Configuration Sections

• [session]: Session timeout, max sessions per user, default isolation level
• [locks]: Lock timeout, deadlock detection interval
• [dump]: Automated dump scheduling, compression settings, retention
• [resource_quotas]: Memory limits, connection limits, query timeouts

API

• Session API: create_session(), destroy_session(), begin_transaction(), commit_transaction(), rollback_transaction()
• Dump API: dump(), restore(), verify_dump(), list_dumps()
• Transaction API: set_isolation_level(), get_isolation_level()

System Views

• pg_stat_activity: Active session monitoring
• pg_locks: Lock information and wait states
• pg_stat_dump_history: Dump operation history

Documentation

• User Guides: In-memory mode guide, dump/restore procedures
• Reference: CLI v3.1 reference, configuration v3.1 reference
• API Documentation: Session management API, lock manager API, dump API
• Architecture: Multi-user transactions architecture with diagrams
• Migration Guide: v3.0 to v3.1 migration guide

Changed

Performance Improvements

• Lock-free concurrent session management (10-100x faster)
• Optimized lock acquisition with adaptive backoff
• Batch dump operations (10,000 rows per batch)
• Streaming restore for large dumps (no full load in memory)
• Early lock release in commit path
• Read-heavy optimization with MVCC snapshots (no read locks)

Behavior Changes (Non-Breaking)

• In-memory mode now warns on shutdown if dirty state exists
• Sessions automatically cleaned up after timeout (default: 1 hour)
• Lock timeouts default to 30 seconds (configurable)
• Dumps default to zstd compression (3x-5x reduction)

Fixed

None (all fixes from v3.0.0 included)

Deprecated

None

Removed

None

Breaking Changes

None. v3.1.0 is 100% backward compatible with v3.0.0.

Security

• Argon2 password hashing for user authentication
• Per-user resource quotas prevent resource exhaustion
• Session timeouts prevent abandoned connections
• Audit logging for session creation/destruction
• Dirty state tracking prevents data loss on shutdown

Migration

From v3.0.0 to v3.1.0:

• No code changes required
• Existing applications work without modification
• New features are opt-in
• Configuration file backward compatible

Known Limitations

1. Session Management: Optional feature, not required for single-user apps
2. Dump Format: .heliodump files not compatible with v3.0.0
3. In-Memory Only: Dump/restore primarily for in-memory mode (persistent mode has WAL)
4. No Distributed Transactions: Single-node transactions only

Performance Metrics

Measured on representative fixtures; reproduce on your own hardware.

Metric	Measured
Concurrent Sessions	10,000+
Transaction Latency (p99)	1-5ms
Lock Acquisition	0.05-1ms
Dump Throughput	66-100MB/s
Restore Throughput	40-66MB/s

---

[3.0.0] - 2025-12-06 - GA Release

Highlights

HeliosDB-Lite v3.0.0 is the first General Availability release, featuring:

• Enhanced REPL: 19 new commands for vectors, agents, and documents
• Comprehensive Configuration: 8 new config sections for AI, RAG, and search
• Feature-Based Help: Intuitive help system organized by functionality

Added

New REPL Commands

• Session Management Commands

  • \sessions - List all agent sessions
  • \session-new <name> - Create a new agent session
  • \session <id> - Show session details
  • \session-delete <id> - Delete an agent session
  • \chat <id> - Interactive chat with session
  • \session-clear <id> - Clear session messages

• Vector Store Commands

  • \vectors - List all vector stores
  • \vector <name> - Show vector store details
  • \vector create <name> <dims> [metric] - Create vector store
  • \vector delete <name> - Delete vector store
  • \vector stats <name> - Show vector store statistics

• Document Collection Commands

  • \collections - List all document collections
  • \collection <name> - Show collection details
  • \docs <collection> - List documents in collection
  • \doc <collection> <id> - Get document details
  • \search-docs <query> - Search across document collections

• Performance & Utility Commands

  • \explain <query> - Show query execution plan
  • \profile <query> - Profile query with detailed timing
  • \telemetry - Show database statistics and health

Help System Improvements

• Feature-based help organization (not version-based)
• New help categories: basics, schema, branching, time-travel, vectors, documents, agents, ai, settings, examples, sql
• Terminal-responsive layout (2-column for narrow, 3-column for wide terminals)
• Integrated examples with each command
• Category-specific help via \h <category>

Configuration Sections

• [ai] - AI feature toggle
• [ai.llm] - LLM provider configuration (OpenAI, Anthropic, Ollama)
• [ai.embedding] - Embedding model configuration
• [rag] - RAG chunking and retrieval settings
• [search] - Full-text search configuration
• [wasm] - WebAssembly execution limits
• [mcp] - Model Context Protocol server settings (planned; no MCP server ships in 3.6.0)
• [storage.tuning] - Advanced storage optimization

Changed

• AI commands (\ai infer, \ai generate, \ai optimize) now use direct database API instead of showing REST API hints
• Improved error messages with helpful suggestions
• Schema analysis provides actionable index recommendations

Fixed

• Help system now organizes by feature instead of version numbers
• Consistent type handling in session and vector commands

[2.5.0] - 2025-12-01

Added

REST API (v1)

• Full HTTP REST API with 14 endpoints via Axum framework
• Branch management endpoints (GET/POST/DELETE /v1/branches)
• Table management endpoints (GET/POST/DELETE /v1/branches/{branch}/tables)
• Data CRUD endpoints (GET/POST/PUT/DELETE /v1/branches/{branch}/tables/{table}/data)
• SQL query execution (POST /v1/branches/{branch}/query)
• SQL statement execution (POST /v1/branches/{branch}/execute)
• Vector similarity search endpoint (POST /v1/branches/{branch}/vector/search)
• Health check endpoint (GET /v1/health)
• Database info endpoint (GET /v1/info)

REST API Models

• Comprehensive request/response models for all endpoints
• QueryRequest/QueryResponse for SQL query results
• ExecuteRequest/ExecuteResponse for DDL/DML operations
• InsertDataRequest/InsertDataResponse for data insertion
• UpdateDataRequest/UpdateDataResponse for data updates
• DeleteDataRequest/DeleteDataResponse for data deletion
• VectorSearchRequest/VectorSearchResponse for similarity search
• Parameterized query support with QueryParameter types
• Pagination support with PaginationInfo

API Middleware

• JWT authentication middleware with configurable secret key
• API key authentication support
• Rate limiting middleware with configurable limits
• Flexible authentication strategies
• Request-scoped user information
• Customizable rate limiting rules

API Server Infrastructure

• Shared database connection across handlers
• Graceful shutdown support
• Configurable bind address and port
• CORS support for cross-origin requests
• Structured error responses

Audit Logging

• Audit logger for comprehensive operation tracking
• DDL event logging (CREATE, ALTER, DROP operations)
• DML event logging (INSERT, UPDATE, DELETE operations)
• Configurable async buffer for high-throughput logging
• Audit log table (__audit_log) for persistent storage

Documentation

• REST API endpoint documentation with examples
• Authentication guide
• Rate limiting configuration guide

Changed

Storage Engine

• Improved branch manager concurrency
• Enhanced table scanning for audit logging

Query Handlers

• Schema inference from query results
• Tuple to JSON row conversion
• DataType string representation for all 17 types

Fixed

• FSST dictionary array size handling (padded to 8 bytes)
• DataType match exhaustiveness in query handler
• Consistent naming of the binary value type (embedded API change, licensed source distribution)

Security

• JWT-based authentication with HS256 algorithm
• API key authentication alternative
• Rate limiting to prevent abuse
• Query parameter validation

[2.2.0] - 2025-11-24

Added

Storage & Transactions

• MVCC snapshot isolation for transactions with proper versioning
• Complete WAL replay logic for crash recovery
• Table name extraction from keys in WAL entries for better observability
• Database statistics tracking (transaction commits, rollbacks, tuple operations)
• Thread-safe atomic counters for all transaction statistics
• Branch parent name lookup capability in branch management
• Snapshot size calculation for capacity planning and monitoring
• Enhanced transaction isolation guarantees (read-your-own-writes, snapshot reads)

Query Optimizer

• Cost-based join algorithm selection (hash join vs nested loop)
• Nested loop join physical operator for small tables and non-equality joins
• Query statistics collection and usage in planner decisions
• Real table statistics (row counts, column distinct values, selectivity)
• Statistics analyzer for collecting and managing table/column statistics
• Selectivity estimation for predicates (equality, range, NULL, AND, OR, NOT)
• Enhanced planner API with cost estimator support
• Verbose mode for debugging query plan decisions
• Explain plan support for all physical operators including nested loop join

PostgreSQL Protocol

• Prepared statement schema derivation from query AST
• Automatic type mapping from HeliosDB types to PostgreSQL OIDs
• Enhanced Parse message handling with schema extraction
• Result schema storage in prepared statements
• Improved Describe message responses with actual column types

Compression

• Comprehensive compression configuration API with fluent builder pattern
• Compression builder for multi-table configuration
• Compression API for statistics and management
• FSST dictionary persistence (export and restore symbol tables)
• Batch compression optimization with pre-allocation and chunking
• Adaptive compression with automatic cost-benefit analysis
• Intelligent sampling strategies (Stratified, BeginningWeighted, Diversity, Auto)
• Compression value count tracking in metadata
• Column-level compression statistics
• Enhanced FSST encoder with multiple compression modes

System Views

• pg_stat_optimizer view with real query statistics
• Enhanced pg_stat_database with transaction statistics
• Enhanced pg_branches with parent branch names
• Enhanced pg_snapshots with actual snapshot sizes
• pg_compression_stats view for compression monitoring

Vector Search

• Vector index length tracking for all metric types (Cosine, InnerProduct, L2)
• len() and is_empty() methods for cosine and inner-product HNSW indexes
• Accurate vector count reporting in multi-metric HNSW indexes

Changed

Storage Engine

• Improved WAL replay with two-pass algorithm (transaction-aware)
• Enhanced transaction handling with snapshot manager integration
• Better error handling in WAL replay (resilient to partial failures)
• Shared storage engine for better concurrency

Query Optimizer

• Planner now uses real statistics instead of hardcoded estimates
• Improved join algorithm selection based on actual cardinality
• Cost estimation now considers table sizes and selectivity
• Physical planner defaults to optimal algorithms per query pattern
• Better query plan quality with accurate row count estimates

Compression

• FSST codec now supports persistent dictionaries
• Improved batch compression performance (10-15% faster at scale)
• Better compression quality with intelligent sampling (1-5% improvement)
• Compression statistics now include value count tracking
• Column compression metadata extended with detailed metrics

PostgreSQL Protocol

• Extended query protocol now returns proper result schemas
• Better client compatibility with automatic schema extraction
• Parse phase includes one-time schema derivation (~160μs overhead)

System Views

• All system views now return real data instead of placeholder values
• Improved accuracy across all statistics views
• Better monitoring and observability capabilities

Fixed

Storage Layer

• Fixed MVCC transaction isolation incomplete implementation
• Fixed WAL replay that only counted entries without applying them
• Fixed table name extraction from storage keys (was always “unknown”)
• Fixed snapshot isolation for concurrent transactions
• Fixed crash recovery

Query Optimizer

• Fixed build validation items in logical expression handling
• Fixed missing cost-based decision logic for join selection
• Fixed hardcoded selectivity estimates (now uses real statistics)

Compression

• Fixed missing compression value count in statistics
• Fixed FSST dictionary serialization (was returning empty vectors)
• Fixed compression metadata propagation

Vector Search

• Fixed vector index length always returning 0
• Fixed multi-metric HNSW index length calculation

System Views

• Fixed pg_stat_optimizer returning placeholder data
• Fixed pg_stat_database missing transaction statistics
• Fixed pg_branches parent lookup
• Fixed pg_snapshots size calculation
• Fixed EXPLAIN statistics usage

Protocol

• Fixed prepared statement schema derivation
• Fixed incomplete extended query protocol support

Type Safety

• Fixed type mismatches in various storage modules
• Improved error handling throughout codebase

Performance

Query Optimizer

• 10-100x speedup for queries with small table joins (nested loop optimization)
• Better query plans with real statistics reduce execution time
• Cost-based selection prevents unnecessary hash table construction
• O(log N) version lookups in MVCC read path

Compression

• 10-15% faster batch compression with pre-allocation
• Better CPU cache utilization with 64-item chunks
• Dictionary serialization/deserialization in ~10-25 μs
• Compression ratios: 2.0-2.5x (emails), 1.8-2.2x (URLs), 1.5-2.0x (mixed strings)
• Adaptive compression prevents wasting resources on incompressible data

WAL & Recovery

• Two-pass WAL replay with progress reporting (every 1000 operations)
• Resilient error handling (continues on non-fatal errors)
• Idempotent replay (safe to replay multiple times)

Statistics

• Negligible overhead for join decision making (~1.6 μs)
• Efficient selectivity estimation with O(1) lookups
• Real-time statistics updates with atomic counters

Deprecated

None

Removed

None

Security

• Enhanced data integrity with MVCC snapshot isolation
• Better crash recovery guarantees with complete WAL replay
• Improved audit trail with accurate table name tracking in WAL

Migration Notes

Upgrading from v2.1.0 to v2.2.0:

1. Statistics Collection: The query optimizer now collects real statistics. No SQL step is needed; HeliosDB Lite has no standalone ANALYZE statement.

2. Compression Configuration: New compression API is opt-in. Existing compressed data remains compatible.

3. WAL Format: Enhanced WAL entries now include table names. Existing WAL files are forward-compatible.

4. System Views: All system views now return real data. Update any queries that relied on placeholder values.

5. API Changes: All changes are backward-compatible. New constructors added, existing code works unchanged.

---

[2.1.0] - 2025-11-XX

Added

• Physical query planner implementation
• Network protocol enhancements
• PostgreSQL wire protocol improvements

Changed

• Improved type safety throughout codebase
• Enhanced system view accuracy

---

[0.1.0-beta] - 2025-11-15

First Beta Release

This is the first public beta release of HeliosDB Lite, a PostgreSQL-compatible embedded database with vector search, encryption, and multi-tenancy.

Added

Core Database Features

• Storage Layer: RocksDB-based LSM-tree storage with MVCC snapshot isolation
• SQL Support: PostgreSQL-compatible SQL engine (95%+ compatibility)
  • Full CRUD operations (CREATE TABLE, INSERT, UPDATE, DELETE, SELECT, DROP TABLE)
  • WHERE clauses with complex expressions
  • Aggregations (COUNT, SUM, AVG, MIN, MAX, COUNT(DISTINCT))
  • GROUP BY, HAVING, ORDER BY, LIMIT, OFFSET, DISTINCT
  • INNER JOIN support (hash join in progress)
  • Transactions with ACID guarantees
  • Snapshot isolation for consistent reads
• Data Types: Complete PostgreSQL type system
  • Numeric: INT2, INT4, INT8, FLOAT4, FLOAT8, NUMERIC, DECIMAL
  • String: TEXT, VARCHAR, CHAR
  • Binary: BYTEA
  • Boolean: BOOL
  • UUID: UUID
  • JSON: JSON, JSONB (bincode-compatible)
  • Array: Array types
  • Vector: VECTOR(n) for embeddings

Security Features

• Transparent Data Encryption (TDE): AES-256-GCM encryption
  • <3% performance overhead with AES-NI
  • Argon2 key derivation (NIST recommended)
  • Per-table encryption support
  • Secure key management (environment, file, in-memory)
  • Automatic nonce generation
  • Memory zeroization for sensitive data
• Audit Logging: Comprehensive audit system
  • Authentication events
  • Data access tracking
  • Schema changes
  • Security events
  • Structured JSON logs
  • Query logging with metadata

Advanced Features

• Vector Search: Built-in HNSW index for AI/ML embeddings
  • Support for 1536-dimension vectors (OpenAI ada-002 compatible)
  • Distance metrics: Cosine, L2 (Euclidean), Inner Product
  • Fast k-NN search (<50ms @ 1M vectors target)
  • Configurable index parameters (M, ef_construction)
  • Serialization support
• JSONB Support: Native JSON binary format
  • Bincode-compatible storage
  • Efficient binary encoding
  • Full JSONB querying (operators in progress)
• Interactive REPL: PostgreSQL-style interactive shell
  • Tab completion for tables, columns, keywords
  • Meta-commands (\d, \dt, ?, \q)
  • Pretty table output
  • Multi-line query support
  • Persistent command history

Network & Protocol

• PostgreSQL Wire Protocol: 85% compatible
  • TCP server with async Tokio
  • SCRAM-SHA-256 authentication
  • Message handling infrastructure
  • Session management
  • Transaction status tracking
  • (Message encoding/decoding completion in progress)
• Server Mode: PostgreSQL-compatible network server
• Embedded Mode: SQLite-style in-process usage
• In-Memory Mode: ACID-compliant RAM-only storage

Developer Experience

• Comprehensive Documentation
  • Rustdoc API documentation
  • Complete GETTING_STARTED.md guide
  • 8 working examples
  • Quick reference guides
  • Troubleshooting documentation

Performance

• Binary Size: Optimized for embedded usage
• Memory Footprint: Efficient with configurable cache
• Encryption Overhead: <3% with AES-NI

Dependencies

Core Storage

• RocksDB 0.22 (LSM-tree storage)
• Apache Arrow 53 (columnar format)
• Apache Parquet 53 (columnar compression)

SQL Processing

• sqlparser-rs 0.53 (PostgreSQL dialect)

Cryptography

• AES-GCM 0.10 (encryption)
• Argon2 0.5 (key derivation)
• Ring 0.17 (cryptographic primitives)
• RustLS 0.23 (TLS 1.3)

Vector Search

• hnsw_rs 0.3 (HNSW index)

Async Runtime

• Tokio 1.40 (async runtime)

Breaking Changes

None (initial release)

Security

• Transparent Data Encryption (TDE) with AES-256-GCM
• SCRAM-SHA-256 authentication
• Secure key derivation with Argon2
• Memory zeroization for sensitive data
• Comprehensive audit logging

Known Limitations (Beta)

1. PostgreSQL Protocol: Message encoding 85% complete (works with basic clients)
2. Hash Joins: Partial implementation (INNER JOIN works, optimizations pending)
3. JSONB Operators: Storage works, query operators in development
4. Multi-Tenancy: Foundation exists, RLS not yet implemented
5. Full-Text Search: Planned for a future release
6. Replication: Single-node only (clustering planned for full HeliosDB)

Migration Path

HeliosDB Lite provides a clear upgrade path to full HeliosDB:

• Data format compatible
• SQL compatibility maintained
• CDC-based migration
• Zero-downtime tenant migration

System Requirements

• Rust: 1.75+ (for building from source)
• Operating Systems: Linux, macOS, Windows
• Architecture: x86_64, ARM64
• Memory: Minimum 512MB, recommended 2GB+
• Disk: Varies by data size (uses compression)

Contributors

This release was made possible by the HeliosDB team and community contributors.

Links

• Documentation: HeliosDB Lite documentation
• Product page: https://www.heliosdb.com/lite.html
• Issues: support@heliosdb.com
• Community: Discord

---

Upgrade Guide

From Source Build to v0.1.0-beta

HeliosDB Lite is distributed to licensed customers as a source distribution that builds the heliosdb-lite binary; see Installation.

If you were using a source build:

1. Stop every process that has the data directory open, then back up the directory.

2. Build the new release in the unpacked source distribution:

   cargo build --release

3. Data format is compatible (no migration needed): open the existing data directory with the new heliosdb-lite binary.

Applications that embed HeliosDB Lite in-process rebuild against the new release (embedded API, licensed source distribution).

API Stability

Beta Stability Guarantees:

• Core API is stable
• SQL syntax is stable (PostgreSQL compatible)
• Data format is forward-compatible
• Configuration format is stable

May Change Before v1.0:

• Internal interfaces of the embedded API (licensed source distribution)
• Experimental features (documented as such)
• Performance tuning parameters