HeliosDB Lite Deployment Modes Guide

Overview

HeliosDB Lite supports 4 distinct deployment modes, each optimized for different use cases. This guide helps you choose the right mode for your application and configure it for optimal performance.

Latency and throughput figures in this guide are indicative, measured on representative fixtures; reproduce on your own hardware.

Quick Comparison

Feature	Embedded	Memory-Only	Server	Hybrid (recommended)
Network Access	No	No	Yes	Yes
Persistent Storage	Yes	No	Yes	Yes
Memory Cache	Partial (basic)	Yes (full)	Partial (basic)	Yes (configurable)
Multi-Client	No	No	Yes	Yes
PostgreSQL Protocol	No	No	Yes	Yes
Performance	Good	Excellent	Good	Excellent
Durability	Yes	No	Yes	Yes
Best For	Single-process apps	Testing, CI/CD	Basic server	Production

Statement latency does not depend on the mode (internal tests, HeliosDB Lite, same statements in each mode): CREATE TABLE 4.3 ms, INSERT 3.6 ms, SELECT 0.15 ms. Startup is instant for embedded and memory-only use, about 500 ms for the server and about 1 s for hybrid mode.

---

Mode 1: Embedded Mode

Description

SQLite-style in-process database with persistent storage. The database runs within your application’s process.

Architecture

┌─────────────────────────────┐
│   Your Application          │
│  ┌────────────────────────┐ │
│  │ HeliosDB Lite (Embed)  │ │
│  │  ┌──────────────────┐  │ │
│  │  │  Storage Engine  │  │ │
│  │  └────────┬─────────┘  │ │
│  └───────────┼─────────────┘ │
└──────────────┼───────────────┘
               │
               ▼
        ┌──────────────┐
        │  Disk Files  │
        │ (RocksDB)    │
        └──────────────┘

Configuration

CLI:

# REPL with persistent storage
heliosdb-lite repl -d ./mydb

Embedding the engine in your own application process is available to licensed customers with the source distribution; contact sales@heliosdb.com.

Use Cases

• Desktop applications (Electron, Tauri)
• Mobile apps (via FFI bindings)
• Edge devices and IoT
• Single-process microservices
• Command-line tools
• Embedded systems

Pros

• Simple deployment (single process)
• Zero network overhead
• Low latency
• No port conflicts
• Easy to backup (just copy files)

Cons

• Single-process access only
• Cannot scale horizontally
• No network access for monitoring tools

Performance Characteristics

• Latency: 0.1-1ms (no network overhead)
• Throughput: 10,000-50,000 ops/sec (depends on disk)
• Memory Usage: Low (minimal caching)

---

Mode 2: Memory-Only Mode

Description

ACID-compliant RAM-only storage. By default, data is lost when the process terminates; from the command prompt, add --dump-on-shutdown to start --memory or repl --memory to write a graceful-exit dump.

Architecture

┌─────────────────────────────┐
│   Your Application          │
│  ┌────────────────────────┐ │
│  │ HeliosDB Lite (Memory) │ │
│  │  ┌──────────────────┐  │ │
│  │  │   RAM Storage    │  │ │
│  │  │  (Temporary Dir) │  │ │
│  │  └──────────────────┘  │ │
│  └────────────────────────┘ │
└─────────────────────────────┘
         (No disk writes)

Configuration

CLI:

# REPL in memory-only mode
heliosdb-lite repl --memory

Use Cases

• Testing and CI/CD pipelines
• Temporary data processing
• Fast analytics on ephemeral data
• Development and debugging
• Session storage (short-lived)
• Cache with SQL interface

Pros

• Extremely fast (no disk I/O)
• No cleanup required
• Reproducible test environments
• Zero disk space usage

Cons

• Data is RAM-only unless you enable a shutdown dump with --dump-on-shutdown
• Limited by RAM size
• Not suitable for production data

Performance Characteristics

• Latency: 0.01-0.1ms (RAM-only)
• Throughput: 100,000+ ops/sec
• Memory Usage: High (all data in RAM)

---

Mode 3: Server Mode

Description

PostgreSQL-compatible network server with persistent storage. Enables multi-client access via PostgreSQL wire protocol.

Architecture

┌──────────┐  ┌──────────┐  ┌──────────┐
│  Client  │  │  Client  │  │  Client  │
│  (psql)  │  │  (App)   │  │  (pgAdmin)│
└────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │              │
     └─────────────┼──────────────┘
                   │ PostgreSQL Wire Protocol
              ┌────▼─────────────────┐
              │  HeliosDB Server     │
              │  ┌────────────────┐  │
              │  │ Storage Engine │  │
              │  └────────┬───────┘  │
              └───────────┼──────────┘
                          │
                          ▼
                   ┌──────────────┐
                   │  Disk Files  │
                   └──────────────┘

Configuration

Config file (heliosdb.toml):

[storage]
path = "./heliosdb-data"
memory_only = false
cache_size = 134217728  # 128 MB (basic cache)

[server]
listen_addr = "127.0.0.1"
port = 5432
max_connections = 100
tls_enabled = false

CLI:

export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"
# Initialize database
heliosdb-lite init ./mydb

# Start server
heliosdb-lite start --data-dir ./mydb --port 5432

# Or with config file
heliosdb-lite start --data-dir ./mydb --config heliosdb.toml

Connect with clients:

# psql
psql postgresql://localhost:5432/mydb

# pgAdmin, DBeaver, etc.
# Connection string: postgresql://localhost:5432/mydb

Use Cases

• Web applications (multi-client)
• Microservices architecture
• PostgreSQL migration (95%+ compatible)
• Multi-language applications
• Remote database access
• Monitoring and admin tools

Pros

• Multi-client support
• PostgreSQL wire protocol compatibility
• Works with existing PostgreSQL tools
• Network-accessible
• Familiar server model

Cons

• Network latency overhead
• Port management required
• More complex deployment
• Basic caching (not optimized)

Performance Characteristics

• Latency: 1-10ms (network overhead)
• Throughput: 5,000-20,000 ops/sec (network-bound)
• Memory Usage: Moderate

---

Mode 4: Hybrid Mode (Server + Memory Cache) (recommended)

Description

Recommended for production. Combines server mode with persistent storage AND configurable memory caching for optimal performance.

Architecture

┌──────────┐  ┌──────────┐  ┌──────────┐
│  Client  │  │  Client  │  │  Client  │
└────┬─────┘  └────┬─────┘  └────┬─────┘
     │             │              │
     └─────────────┼──────────────┘
                   │ PostgreSQL Wire Protocol
              ┌────▼──────────────────────┐
              │  HeliosDB Server (Hybrid) │
              │  ┌──────────────────────┐ │
              │  │   Memory Cache       │ │
              │  │   (512MB - 4GB)      │ │
              │  ├──────────────────────┤ │
              │  │   Storage Engine     │ │
              │  └──────────┬───────────┘ │
              └─────────────┼─────────────┘
                            │
                            ▼
                     ┌──────────────┐
                     │  Disk Files  │
                     │  (RocksDB)   │
                     └──────────────┘

Why Hybrid Mode?

Hybrid mode provides:

1. Network Access: Multi-client support via PostgreSQL protocol
2. Persistent Storage: Data durability with RocksDB
3. Memory Cache: Configurable in-memory cache for hot data
4. Performance: Best of both worlds

This is the only mode that combines all these benefits.

Configuration

Recommended config (heliosdb.toml):

[storage]
# Persistent storage path
path = "./heliosdb-data"

# NOT memory-only (enable disk persistence)
memory_only = false

# Memory cache size (key configuration!)
# Adjust based on available RAM:
cache_size = 536870912      # 512 MB (for 4-8 GB RAM systems)
# cache_size = 1073741824   # 1 GB (for 8-16 GB RAM systems)
# cache_size = 2147483648   # 2 GB (for 16-32 GB RAM systems)
# cache_size = 4294967296   # 4 GB (for 32+ GB RAM systems)

# Enable compression for better cache efficiency
compression = "zstd"        # or "lz4" for faster compression

# Write-ahead log for crash recovery
wal_enabled = true

[server]
listen_addr = "127.0.0.1"   # Use "0.0.0.0" for external access
port = 5432
max_connections = 100

# Optional: Enable TLS for production
tls_enabled = false
# tls_cert_path = "/path/to/cert.pem"
# tls_key_path = "/path/to/key.pem"

[performance]
# Use all CPU cores
worker_threads = 8          # Adjust based on CPU count

# Enable SIMD acceleration
simd_enabled = true

# Enable parallel query execution
parallel_query = true

# Query timeout (seconds)
query_timeout_secs = 300

[encryption]
# Optional: Enable encryption at rest
enabled = false
algorithm = "Aes256Gcm"
key_source = { Environment = "HELIOSDB_ENCRYPTION_KEY" }

Start server:

export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"
# Using config file (recommended; cache_size and max_connections live in the file)
heliosdb-lite start --data-dir ./heliosdb-data --config heliosdb.toml

# Or with CLI flags only (default cache and connection limits)
heliosdb-lite start --data-dir ./heliosdb-data --port 5432

Cache Size Tuning

How to choose cache_size:

System RAM	Recommended cache_size	Percentage
4 GB	512 MB (536870912)	~12%
8 GB	1 GB (1073741824)	~12%
16 GB	2 GB (2147483648)	~12%
32 GB	4 GB (4294967296)	~12%
64 GB+	8 GB+ (8589934592)	~12-25%

Rule of thumb: Allocate 10-25% of available RAM to cache, leaving room for OS and application memory.

Monitoring cache efficiency:

-- Check cache hit rate (planned feature)
SELECT cache_hit_rate FROM system_stats;

Use Cases

• Production web applications (high traffic)
• High-performance microservices
• Multi-tenant SaaS applications
• Real-time analytics
• API backends with hot data
• Content management systems
• E-commerce platforms

Pros

• Best performance (memory cache + persistence)
• Multi-client support (network server)
• Data durability (persistent storage)
• Configurable (tune for your workload)
• Scalable (within single node)

Cons

• Higher memory usage (but configurable)
• More complex configuration (but flexible)
• Requires tuning for optimal performance

Performance Characteristics

• Latency:
  • Cache hit: 0.5-2ms (fast)
  • Cache miss: 5-15ms (disk read + network)
• Throughput: 20,000-100,000 ops/sec (depends on cache hit rate)
• Memory Usage: Configurable (512MB - 8GB+)
• Cache Hit Rate: 80-95% (for typical workloads)

Performance Optimization

1. Monitor your workload:

-- Identify hot tables/queries
EXPLAIN ANALYZE SELECT * FROM users WHERE active = true;

2. Tune cache size:

• Start with 512 MB (conservative)
• Monitor cache hit rate
• Increase if hit rate < 80%
• Decrease if memory pressure

3. Use compression:

# Zstd: Best compression (slower)
compression = "zstd"

# LZ4: Faster compression (larger cache footprint)
compression = "lz4"

4. Enable all performance features:

[performance]
simd_enabled = true        # SIMD acceleration
parallel_query = true      # Parallel execution
worker_threads = 8         # Use all cores

---

Deployment Comparison Matrix

By Use Case

Use Case	Recommended Mode	Why?
Desktop app	Embedded	Single-process, simple
Mobile app	Embedded	Offline-first, local storage
Testing/CI	Memory-Only	Fast, reproducible
Development	Memory-Only or Embedded	Quick iteration
Web app (low traffic)	Server	Multi-client, basic needs
Web app (production)	Hybrid (recommended)	Performance + durability
Microservices	Hybrid (recommended)	Multi-client, high performance
Analytics	Hybrid or Memory-Only	Large cache or all-in-memory
Edge computing	Embedded	Local-first, offline
PostgreSQL migration	Server or Hybrid	Wire protocol compatible

By Performance Needs

Requirement	Mode	Rationale
Lowest latency	Memory-Only	No disk I/O
Best throughput (single-client)	Memory-Only	RAM-only
Best throughput (multi-client)	Hybrid (recommended)	Cache + parallel
Highest durability	Embedded or Hybrid	Persistent storage
Network access	Server or Hybrid	Wire protocol
operationally hardened	Hybrid (recommended)	All features

By Resource Constraints

Constraint	Mode	Notes
Low memory (< 2GB)	Embedded	Minimal caching
No disk space	Memory-Only	Temporary only
No network access	Embedded or Memory-Only	Local only
High memory (16GB+)	Hybrid (recommended)	Large cache
Multi-core CPU	Hybrid (recommended)	Parallel queries

---

Migration Between Modes

Embedded → Hybrid

Why migrate:

• Need multi-client access
• Want better performance
• Scaling beyond single process

Steps:

1. Create config file:

[storage]
path = "./existing-data"  # Point to existing data
memory_only = false
cache_size = 1073741824   # 1 GB cache

[server]
port = 5432
max_connections = 100

2. Start server:

export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"
heliosdb-lite start --data-dir ./existing-data --config heliosdb.toml

3. Update application to connect over the PostgreSQL port:

use postgres::Client;
let mut client = Client::connect(
    "postgresql://localhost:5432/mydb",
    NoTls
)?;

Memory-Only → Hybrid

Why migrate:

• Need persistence
• Moving from development to production

Steps:

1. Export schema and data:

-- In memory-only instance
\d  -- Note all tables
-- Export data (manually or via pg_dump equivalent)

2. Set up Hybrid mode:

export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"
# Create config (as shown above)
heliosdb-lite start --data-dir ./heliosdb-data --config heliosdb.toml

3. Import data:

-- Recreate schema
CREATE TABLE ...;

-- Import data
INSERT INTO ...;

Server → Hybrid

No migration is needed; tune the cache:

[storage]
# Increase cache size
cache_size = 2147483648  # 2 GB (was 128 MB)

Restart server and you’re in optimized Hybrid mode.

---

Configuration Best Practices

Development

# Quick iteration, minimal resources
[storage]
path = "./dev-data"
memory_only = false
cache_size = 134217728    # 128 MB (small cache)
compression = "lz4"       # Fast compression

[server]
listen_addr = "127.0.0.1" # Localhost only
port = 5432
max_connections = 10      # Few connections

Staging

# Production-like, but smaller
[storage]
path = "./staging-data"
memory_only = false
cache_size = 536870912    # 512 MB
compression = "zstd"      # Production compression

[server]
listen_addr = "0.0.0.0"   # Network access
port = 5432
max_connections = 50

[encryption]
enabled = true            # Test encryption

Production (Hybrid) (recommended)

# Optimized for performance and durability
[storage]
path = "/var/lib/heliosdb"
memory_only = false
cache_size = 2147483648   # 2 GB (tune based on workload)
compression = "zstd"
wal_enabled = true

[server]
listen_addr = "0.0.0.0"
port = 5432
max_connections = 100
tls_enabled = true
tls_cert_path = "/etc/heliosdb/cert.pem"
tls_key_path = "/etc/heliosdb/key.pem"

[performance]
worker_threads = 16       # Use all cores
simd_enabled = true
parallel_query = true
query_timeout_secs = 300

[encryption]
enabled = true
algorithm = "Aes256Gcm"
key_source = { Environment = "HELIOSDB_ENCRYPTION_KEY" }

# Built-in audit logging is planned: the [audit] section is accepted
# but has no effect yet.

---

Monitoring and Diagnostics

Check Current Mode

From logs:

export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"
# Server logs show mode at startup
heliosdb-lite start --data-dir /var/lib/heliosdb --config heliosdb.toml

# Output:
# [INFO] Starting HeliosDB Lite in Hybrid Mode
# [INFO] Cache size: 512 MB
# [INFO] Persistent storage: /var/lib/heliosdb
# [INFO] Server listening on 0.0.0.0:5432

From configuration:

-- Query configuration (planned feature)
SELECT * FROM system_config;

Performance Metrics

Key metrics to monitor:

• Cache hit rate (target: > 80%)
• Query latency (p50, p95, p99)
• Disk I/O (lower is better)
• Memory usage (should be < cache_size + overhead)
• Connection count (should be < max_connections)

---

Summary

Mode	Key Feature	Best For
Embedded	Simple, local	Desktop apps, edge devices
Memory-Only	Fast, ephemeral	Testing, temporary data
Server	Network, persistent	Basic multi-client needs
Hybrid (recommended)	All features	Production applications

Recommendation for most users: Start with Hybrid mode for production. It provides the best balance of performance, durability, and scalability.

---

Additional Resources

• Performance Tuning (coming soon)
• REST API

Questions?

• Support: support@heliosdb.com
• Community: Discord
• Discord: https://discord.gg/yTykuUrFXc