--- title: "HeliosDB Configuration Guide" --- # HeliosDB Configuration Guide > **Container images:** HeliosDB Full images are not published on Docker Hub or any other public registry. Images and Helm charts are provided through private registry access during onboarding: contact [sales@heliosdb.com](mailto:sales@heliosdb.com). The examples on this page use `heliosdb-full:`, an image built locally from the licensed source distribution (`docker build -t heliosdb-full: .`); replace it with the image reference you receive if you use the private registry. ## Overview HeliosDB provides a comprehensive configuration system supporting TOML files, environment variables, and CLI arguments. This guide covers all configuration options for HeliosDB's advanced features including GPU acceleration, multi-region deployment, autoscaling, and CDC. ## Table of Contents - [Configuration Sources](#configuration-sources) - [Configuration File Structure](#configuration-file-structure) - [GPU Configuration](#gpu-configuration) - [Multi-Region Configuration](#multi-region-configuration) - [Autoscaling Configuration](#autoscaling-configuration) - [CDC Configuration](#cdc-configuration) - [Storage Configuration](#storage-configuration) - [Network Configuration](#network-configuration) - [Security Configuration](#security-configuration) - [Environment Variables](#environment-variables) - [Best Practices](#best-practices) - [Examples](#examples) - [Troubleshooting](#troubleshooting) ## Configuration Sources HeliosDB loads configuration in the following priority order (highest to lowest): 1. **Environment Variables** - Runtime overrides (prefixed with `HELIOSDB_`) 2. **TOML Configuration File** - Primary configuration source 3. **Default Values** - Built-in sensible defaults ## Configuration File Structure The configuration file uses TOML format with the following top-level sections: ```toml [storage] # Storage engine configuration [network] # Network and communication settings [metadata] # Raft consensus configuration [vector] # Vector index configuration [security] # Security and encryption settings [gpu] # GPU acceleration settings [multi_region] # Multi-region deployment [autoscale] # Autoscaling configuration [cdc] # Change Data Capture (planned connectors) ``` ## GPU Configuration :::caution[GPU acceleration: builds on request] GPU acceleration (NVIDIA CUDA) is available in HeliosDB Full builds on request and is not enabled in the public release build; AMD ROCm support is planned. In the public release build, queries run on the CPU and the GPU settings and commands on this page have no effect. Contact [sales@heliosdb.com](mailto:sales@heliosdb.com) about a CUDA-enabled build. ::: Enable GPU acceleration for 10-100x speedup on analytical queries. ### Configuration Options ```toml [gpu] # Enable GPU acceleration enabled = true # GPU memory limit in bytes (8GB = 8589934592) memory_limit = 8589934592 # GPU device ID (0-indexed) device_id = 0 # Fallback to CPU if GPU unavailable fallback_to_cpu = true # Enable CUDA unified memory enable_unified_memory = true # GPU vendor: "nvidia", "amd", or "intel" # vendor = "nvidia" # Auto-detect if not specified # Minimum data size to trigger GPU execution (100MB) min_data_size_bytes = 104857600 # Enable GPU for specific operations enable_aggregations = true enable_joins = true enable_vector_ops = true ``` ### GPU Vendor Support - **NVIDIA**: CUDA 11.0+ (Tesla T4, V100, A100) - **AMD**: ROCm/HIP 5.0+ (MI100, MI250X) - **Intel**: Level Zero API (experimental) ### Environment Variables ```bash export HELIOSDB_GPU_ENABLED=true export HELIOSDB_GPU_DEVICE_ID=0 export HELIOSDB_GPU_MEMORY_LIMIT=8589934592 export HELIOSDB_GPU_FALLBACK_TO_CPU=true ``` ### Performance Tuning **Memory Limit**: Set based on your GPU VRAM: - 8GB GPUs: `8589934592` (8GB) - 16GB GPUs: `17179869184` (16GB) - 24GB GPUs: `25769803776` (24GB) **Min Data Size**: Adjust based on overhead: - Small queries (<100MB): CPU is faster - Large queries (>100MB): GPU provides speedup - Optimal threshold: 100-500MB depending on GPU ## Multi-Region Configuration Deploy HeliosDB across multiple geographic regions with automatic replication and failover. ### Configuration Options ```toml [multi_region] # Enable multi-region deployment enabled = true # Primary region identifier primary_region = "us-west" # Replication mode replication_mode = "active_active" # Options: active_passive, active_active, multi_master # Conflict resolution strategy conflict_resolution = "last_write_wins" # Options: last_write_wins, first_write_wins, custom # Enable compression for cross-region traffic compression = true # Enable encryption for cross-region traffic encryption = true # Maximum replication lag in milliseconds max_lag_ms = 5000 # Enable automatic failover auto_failover = true # Region definitions [[multi_region.regions]] id = "us-west" endpoint = "https://us-west.heliosdb.example.com" datacenter = "California" is_primary = true [[multi_region.regions]] id = "us-east" endpoint = "https://us-east.heliosdb.example.com" datacenter = "Virginia" is_primary = false [[multi_region.regions]] id = "eu-central" endpoint = "https://eu-central.heliosdb.example.com" datacenter = "Frankfurt" is_primary = false ``` ### Replication Modes **Active-Passive**: - Single primary region accepts writes - Replicas serve read-only queries - Lowest complexity, eventual consistency **Active-Active**: - Multiple regions accept writes - Automatic conflict resolution - Higher availability, lower write latency **Multi-Master**: - All regions accept writes - Advanced conflict resolution required - Maximum availability and performance ### Conflict Resolution Strategies - **Last Write Wins (LWW)**: Latest timestamp wins conflicts - **First Write Wins (FWW)**: First write wins conflicts - **Custom**: Application-defined resolution logic ### Environment Variables ```bash export HELIOSDB_MULTIREGION_ENABLED=true export HELIOSDB_MULTIREGION_PRIMARY_REGION=us-west ``` ## Autoscaling Configuration Automatically scale compute resources based on workload demand. ### Configuration Options ```toml [autoscale] # Enable autoscaling enabled = true # Minimum compute units (0 = scale-to-zero) min_cu = 10.0 # Maximum compute units max_cu = 1000.0 # Target CPU utilization percentage target_cpu_percent = 70.0 # Scale up when CPU exceeds threshold scale_up_threshold = 80.0 # Scale down when CPU drops below threshold scale_down_threshold = 30.0 # Cooldown period between scaling decisions (seconds) cooldown_period_seconds = 60 # Scale to zero after idle period (seconds) scale_to_zero_idle_seconds = 300 # Scale step percentage (50 = scale by 50% each time) scale_step_percent = 50.0 # Damping factor to prevent oscillation (0.0-1.0) damping_factor = 0.8 [autoscale.horizontal] # Enable horizontal scaling (add/remove nodes) enabled = true # Minimum number of nodes min_nodes = 1 # Maximum number of nodes max_nodes = 10 # Maximum CUs per node node_max_cu = 100.0 # CPU threshold for scaling out (adding nodes) scale_out_cpu_threshold = 85.0 # CPU threshold for scaling in (removing nodes) scale_in_cpu_threshold = 20.0 # Drain timeout when removing nodes (seconds) drain_timeout_seconds = 120 ``` ### Compute Units (CU) 1 CU = 1 vCPU + 4GB RAM **Sizing Guidelines**: - **Small workloads**: 10-50 CU - **Medium workloads**: 50-200 CU - **Large workloads**: 200-1000 CU - **Enterprise**: 1000+ CU ### Scaling Strategies **Vertical Scaling**: - Adjusts CU allocation per node - Fast response (seconds) - Limited by `max_cu` **Horizontal Scaling**: - Adds/removes entire nodes - Slower response (minutes) - Better for sustained load ### Environment Variables ```bash export HELIOSDB_AUTOSCALE_ENABLED=true export HELIOSDB_AUTOSCALE_MIN_CU=10 export HELIOSDB_AUTOSCALE_MAX_CU=1000 export HELIOSDB_AUTOSCALE_TARGET_CPU_PERCENT=70 ``` ## CDC Configuration :::caution[Planned: CDC connectors] CDC connectors are planned. The current server does not act on the `[cdc]` section or on the `HELIOSDB_CDC_*` environment variables; they describe the planned connector configuration. The in-memory change-event log that `SHOW CDC EVENTS` reads needs no configuration. ::: Planned configuration for capturing database changes and streaming them to Kafka or Kinesis. ### Configuration Options ```toml [cdc] # Enable CDC enabled = true # Output format output_format = "avro" # Options: json, avro, protobuf # Buffer size for CDC events buffer_size = 10000 # Flush interval for batching events flush_interval = "5s" # Humantime format: "5s", "1m", "100ms" # Checkpoint interval (number of events) checkpoint_interval = 1000 # Enable compression for CDC streams compression = true # Maximum lag before warning (seconds) max_lag_seconds = 300 # Kafka sink configuration [cdc.kafka] bootstrap_servers = ["localhost:9092"] topic = "heliosdb-cdc" ssl = true sasl = true compression_type = "snappy" # Options: none, gzip, snappy, lz4, zstd # Alternative: Kinesis sink # [cdc.kinesis] # region = "us-west-2" # stream_name = "heliosdb-cdc-stream" # enhanced_fan_out = true ``` ### Output Formats **JSON**: - Human-readable - Larger size - Easy debugging **Avro**: - Schema evolution - Compact binary format - Best performance **Protobuf**: - Type-safe - Compact binary format - Cross-language support ### Sinks **Kafka**: - High throughput - Guaranteed ordering - Mature ecosystem **AWS Kinesis**: - Native AWS integration - Serverless - Enhanced fan-out support ### Environment Variables ```bash export HELIOSDB_CDC_ENABLED=true export HELIOSDB_CDC_BUFFER_SIZE=10000 ``` ## Storage Configuration Core storage engine settings. ```toml [storage] data_dir = "/var/lib/heliosdb" memtable_size_mb = 64 sstable_size_mb = 128 compaction_strategy = "SizeTiered" # or "Leveled" bloom_filter_bits_per_key = 10 gc_grace_seconds = 864000 # 10 days ``` ## Network Configuration Network and communication settings. ```toml [network] listen_addr = "0.0.0.0:6543" rdma_enabled = false max_message_size_mb = 64 ``` ## Security Configuration Encryption and key management. ```toml [security] tde_enabled = false tde_algorithm = "Aes256Gcm" # Options: Aes256Gcm, Aes256Ctr, ChaCha20Poly1305 key_storage_path = "/var/lib/heliosdb/keys" default_mek_id = "master_key_001" key_cache_limit = 1000 hsm_enabled = false # KMS Provider (AWS example) # [security.kms_provider.Aws] # region = "us-west-2" # key_id = "arn:aws:kms:..." ``` ## Environment Variables All configuration options can be overridden via environment variables using the format: ``` HELIOSDB_
_= ``` ### Examples ```bash # GPU Configuration export HELIOSDB_GPU_ENABLED=true export HELIOSDB_GPU_DEVICE_ID=1 export HELIOSDB_GPU_MEMORY_LIMIT=17179869184 # 16GB # Multi-Region Configuration export HELIOSDB_MULTIREGION_ENABLED=true export HELIOSDB_MULTIREGION_PRIMARY_REGION=eu-central # Autoscaling Configuration export HELIOSDB_AUTOSCALE_ENABLED=true export HELIOSDB_AUTOSCALE_MIN_CU=20 export HELIOSDB_AUTOSCALE_MAX_CU=500 # CDC Configuration (planned connectors; not read by the current server) export HELIOSDB_CDC_ENABLED=true export HELIOSDB_CDC_BUFFER_SIZE=20000 # Storage Configuration export HELIOSDB_STORAGE_DATA_DIR=/mnt/fast-ssd/heliosdb export HELIOSDB_STORAGE_MEMTABLE_SIZE_MB=128 # Network Configuration export HELIOSDB_NETWORK_LISTEN_ADDR=0.0.0.0:7654 export HELIOSDB_NETWORK_RDMA_ENABLED=true ``` ## Best Practices ### Development Environment ```toml [gpu] enabled = false # Use CPU for development [autoscale] enabled = false # Fixed resources [multi_region] enabled = false # Single region [cdc] enabled = false # Not needed for dev ``` ### Production Environment ```toml [gpu] enabled = true fallback_to_cpu = true # Ensure availability [autoscale] enabled = true min_cu = 50 # Ensure minimum capacity max_cu = 2000 [multi_region] enabled = true auto_failover = true encryption = true [cdc] enabled = true compression = true ``` ### Performance Optimization 1. **GPU**: Enable for analytical workloads (OLAP) 2. **Autoscaling**: Set aggressive thresholds for cost savings 3. **Multi-Region**: Use active-passive for read-heavy workloads 4. **CDC** (planned connectors): Use Avro format for best performance ### Security Hardening 1. Enable TDE for data-at-rest encryption 2. Enable encryption for multi-region traffic 3. Use KMS for key management 4. Enable HSM for highest security ## Examples ### Minimal Configuration ```toml [storage] data_dir = "/var/lib/heliosdb" [network] listen_addr = "0.0.0.0:6543" ``` ### GPU-Accelerated Analytics ```toml [gpu] enabled = true memory_limit = 17179869184 # 16GB enable_aggregations = true enable_joins = true fallback_to_cpu = true ``` ### Multi-Region HA Setup ```toml [multi_region] enabled = true primary_region = "us-west" replication_mode = "active_active" conflict_resolution = "last_write_wins" auto_failover = true [[multi_region.regions]] id = "us-west" endpoint = "https://us-west.db.example.com" is_primary = true [[multi_region.regions]] id = "eu-central" endpoint = "https://eu.db.example.com" is_primary = false ``` ### Serverless with Scale-to-Zero ```toml [autoscale] enabled = true min_cu = 0 # Scale to zero max_cu = 500 scale_to_zero_idle_seconds = 300 # 5 minutes ``` ### CDC to Kafka (planned) ```toml [cdc] enabled = true output_format = "avro" buffer_size = 10000 flush_interval = "5s" [cdc.kafka] bootstrap_servers = ["kafka1:9092", "kafka2:9092", "kafka3:9092"] topic = "heliosdb-changes" ssl = true compression_type = "snappy" ``` ## Troubleshooting ### Configuration Not Loading **Problem**: Configuration file not found **Solution**: ```bash # Verify file exists ls -la /etc/heliosdb/heliosdb.toml # Check permissions chmod 644 /etc/heliosdb/heliosdb.toml # Use explicit path export HELIOSDB_CONFIG_PATH=/path/to/config.toml ``` ### GPU Not Detected **Problem**: GPU acceleration not working **Solution**: ```bash # Check GPU availability nvidia-smi # NVIDIA rocm-smi # AMD # Verify device ID export HELIOSDB_GPU_DEVICE_ID=0 # Enable CPU fallback export HELIOSDB_GPU_FALLBACK_TO_CPU=true ``` ### Autoscaling Not Working **Problem**: Cluster not scaling **Solution**: ```toml # Check thresholds [autoscale] scale_up_threshold = 80.0 scale_down_threshold = 30.0 cooldown_period_seconds = 60 # Reduce for faster response # Verify min/max are different min_cu = 10.0 max_cu = 100.0 # Must be > min_cu ``` ### CDC Lag Increasing (planned connectors) **Problem**: CDC falling behind **Solution**: ```toml # Increase buffer size buffer_size = 50000 # Reduce flush interval flush_interval = "1s" # Enable compression compression = true ``` ## Configuration Validation HeliosDB automatically validates configuration on startup. Common validation errors: - `max_cu must be greater than min_cu` - `gpu.memory_limit must be > 0 when GPU is enabled` - `multi_region.regions cannot be empty when enabled` - `scale_up_threshold must be > scale_down_threshold` Check logs for validation errors: ```bash tail -f /var/log/heliosdb/heliosdb.log | grep -i "config" ``` ## Support For configuration assistance: - Support: [support@heliosdb.com](mailto:support@heliosdb.com) - Documentation: https://www.heliosdb.com/docs/full/ - Community: [Discord](https://discord.gg/yTykuUrFXc)