Data Compression

HeliosDB Lite Solution

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                 HeliosDB-Lite Data Compression Stack                     │
├─────────────────────────────────────────────────────────────────────────┤
│  SQL Layer: CREATE TABLE with CODEC options, Transparent INSERT/SELECT  │
├─────────────────────────────────────────────────────────────────────────┤
│  Per-Column Compression: FSST (Text) │ ALP (Numeric) │ AUTO │ None      │
├─────────────────────────────────────────────────────────────────────────┤
│  SIMD Acceleration (AVX2/NEON) │ Dictionary Manager │ Compression Stats │
├─────────────────────────────────────────────────────────────────────────┤
│  Storage-Level Compression (Optional): Zstd │ LZ4 │ Snappy              │
├─────────────────────────────────────────────────────────────────────────┤
│              LSM-Tree Storage Engine (RocksDB-based)                     │
└─────────────────────────────────────────────────────────────────────────┘

Concrete Examples with Code, Config & Architecture

Example 1: Log Management System - Embedded Configuration

Architecture:

Microservices (50 instances)
    ↓
Log Aggregator (Fluentd/Vector)
    ↓
HeliosDB-Lite Embedded (in-process)
    ↓
FSST-Compressed Log Storage (LSM-Tree)
    ↓
Query API (REST/gRPC) → Search Dashboard

Configuration (heliosdb.toml):

# HeliosDB-Lite configuration for log compression
[database]
path = "/var/lib/heliosdb/logs.db"
memory_limit_mb = 2048
enable_wal = true
page_size = 16384  # Larger pages for better compression

[compression]
enabled = true
# Automatic codec selection based on column types
adaptive_compression = true
min_compression_ratio = 1.2
# Minimum data size to trigger compression (10KB)
min_data_size = 10240

[compression.fsst]
# Enable FSST for string columns (log messages, stack traces, URLs)
enabled = true
# Sample size for dictionary training (10K rows)
training_sample_size = 10000
# Dictionary cache size (max 100 dictionaries in memory)
dictionary_cache_size = 100

[compression.alp]
# Enable ALP for numeric columns (timestamps, response times, counts)
enabled = true

[storage]
block_compression = "zstd"
block_compression_level = 3  # Balanced compression (1-9)

[monitoring]
metrics_enabled = true
verbose_logging = false

[performance]
# SIMD acceleration auto-detected (AVX2 on x86_64)
simd_enabled = true

Example 3: Content Management System - Docker Deployment

Scenario: SaaS content platform storing user-generated articles, blog posts, and comments for 10K customers, each with 500-5000 content items (5M total documents averaging 2KB text each, 10GB uncompressed).

Docker Deployment (Dockerfile):

FROM rust:1.75-slim as builder

WORKDIR /app

# Copy source
COPY . .

# Build HeliosDB-Lite CMS application
RUN cargo build --release --features compression

# Runtime stage
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y \
    ca-certificates \
    libssl3 \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/target/release/cms-api /usr/local/bin/

# Create data volume mount point
RUN mkdir -p /data /config

# Expose HTTP API port
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# Set data directory as volume
VOLUME ["/data"]

ENTRYPOINT ["cms-api"]
CMD ["--config", "/config/heliosdb.toml", "--data-dir", "/data"]

Docker Compose (docker-compose.yml):

version: '3.8'

services:
  cms-api:
    build:
      context: .
      dockerfile: Dockerfile
    image: cms-api:latest
    container_name: cms-api-prod

    ports:
      - "8080:8080"      # HTTP API

    volumes:
      - ./data:/data                              # Persistent database
      - ./config/heliosdb.toml:/config/heliosdb.toml:ro

    environment:
      RUST_LOG: "info,cms_api=debug"
      HELIOSDB_DATA_DIR: "/data"
      HELIOSDB_COMPRESSION: "fsst"  # Enable FSST for text

    restart: unless-stopped

    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 40s

    networks:
      - cms-network

    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

networks:
  cms-network:
    driver: bridge

volumes:
  cms_data:
    driver: local

Configuration for CMS (config/heliosdb.toml):

[server]
host = "0.0.0.0"
port = 8080

[database]
path = "/data/cms.db"
memory_limit_mb = 384
enable_wal = true
page_size = 8192

[compression]
enabled = true
adaptive_compression = true

[compression.fsst]
# Optimize for text content (articles, comments)
enabled = true
training_sample_size = 5000
dictionary_cache_size = 50

[compression.alp]
# Limited numeric data in CMS
enabled = false

[storage]
# Zstd for extra compression on text-heavy workload
block_compression = "zstd"
block_compression_level = 6

[container]
enable_shutdown_on_signal = true
graceful_shutdown_timeout_secs = 30

[monitoring]
metrics_enabled = true

Example 4: Edge IoT Gateway - Constrained Storage Deployment

Edge Device Configuration (heliosdb_edge.toml):

[database]
# Ultra-low resource footprint for embedded
path = "/mnt/flash/iot/sensors.db"
memory_limit_mb = 128        # Limited RAM on edge device
page_size = 4096
enable_wal = true
cache_mb = 32

[compression]
enabled = true
adaptive_compression = true
# Aggressive compression for storage-constrained device

[compression.fsst]
# Compress device IDs, error messages
enabled = true
training_sample_size = 2000
dictionary_cache_size = 20

[compression.alp]
# Essential for numeric sensor data
enabled = true

[storage]
# LZ4 for fast compression/decompression on slow ARM CPU
block_compression = "lz4"
block_compression_level = 1

[retention]
# Automatic cleanup after 45 days
max_age_days = 45
cleanup_interval_hours = 24

[logging]
# Minimal logging for edge devices
level = "warn"
output = "syslog"

Example 5: Cloud Cost Optimization - Multi-Tenant SaaS

Kubernetes Deployment (k8s-cms-deployment.yaml):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: heliosdb-cms
  namespace: production
spec:
  serviceName: heliosdb-cms
  replicas: 3  # HA deployment
  selector:
    matchLabels:
      app: heliosdb-cms
  template:
    metadata:
      labels:
        app: heliosdb-cms
    spec:
      containers:
      - name: heliosdb-cms
        image: heliosdb-cms:<version>
        imagePullPolicy: Always

        ports:
        - containerPort: 8080
          name: http
          protocol: TCP

        env:
        - name: RUST_LOG
          value: "info"
        - name: HELIOSDB_DATA_DIR
          value: "/data"
        - name: HELIOSDB_COMPRESSION
          value: "auto"  # Automatic codec selection
        - name: HELIOSDB_COMPRESSION_LEVEL
          value: "6"  # Balanced

        volumeMounts:
        - name: data
          mountPath: /data
        - name: config
          mountPath: /config
          readOnly: true

        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "1000m"

        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: gp3  # AWS EBS gp3
      resources:
        requests:
          storage: 250Gi  # 625GB / 3 replicas ≈ 210GB + overhead

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: heliosdb-config
  namespace: production
data:
  heliosdb.toml: |
    [database]
    path = "/data/cms.db"
    memory_limit_mb = 768
    enable_wal = true
    page_size = 8192

    [compression]
    enabled = true
    adaptive_compression = true
    min_compression_ratio = 1.3

    [compression.fsst]
    enabled = true
    training_sample_size = 10000
    dictionary_cache_size = 100

    [compression.alp]
    enabled = true

    [storage]
    block_compression = "zstd"
    block_compression_level = 6

---
apiVersion: v1
kind: Service
metadata:
  name: heliosdb-cms
  namespace: production
spec:
  clusterIP: None
  selector:
    app: heliosdb-cms
  ports:
  - port: 8080
    targetPort: 8080
    name: http

---
apiVersion: v1
kind: Service
metadata:
  name: heliosdb-cms-lb
  namespace: production
spec:
  type: LoadBalancer
  selector:
    app: heliosdb-cms
  ports:
  - port: 80
    targetPort: 8080
    name: http

Adoption Strategy

Phase 1: Proof of Concept (Weeks 1-4)

Target: Validate compression ratios and performance on production-like data

Tactics:

• Export sample data from existing database (10-100K rows)
• Import into HeliosDB Lite with automatic compression
• Measure compression ratios, insert/query performance
• Compare storage costs (original vs compressed)

Phase 2: Pilot Deployment (Weeks 5-12)

Tactics:

• Deploy HeliosDB Lite as sidecar or standalone service
• Monitor compression effectiveness, CPU usage, disk I/O
• Collect user feedback on query performance

Phase 3: Full Rollout

Tactics:

• Automated migration scripts with validation
• Comprehensive monitoring (compression ratio, latency, storage)
• Cost tracking dashboard (compare pre/post compression)

Conclusion

Data compression represents a fundamental cost optimization opportunity for organizations struggling with exponential data growth across application logs, time-series metrics, user-generated content, and IoT sensor readings.

Call to Action: Start your compression POC today by deploying HeliosDB Lite on a representative dataset, measuring baseline compression ratios and performance, and projecting annual cost savings.