Database Branching

Executive Summary

HeliosDB Lite's Database Branching feature brings Git-like version control directly to your database, enabling zero-downtime schema migrations, A/B testing, and parallel development workflows. Using copy-on-write (COW) storage with instant branch creation, teams can test schema changes in isolated environments, run parallel experiments, and safely merge updates back to production—all with minimal storage overhead.

---

Problem Being Solved

Core Problem Statement

Database schema changes in production are high-risk operations that require extensive planning, testing windows, and often result in downtime. Traditional migration approaches force teams to choose between speed and safety, making it difficult to iterate quickly while maintaining data integrity. For embedded databases powering edge devices, microservices, or mobile applications, the lack of branching capabilities means every schema change risks breaking production or requires complex versioning schemes.

Who Suffers Most

• Data Analysts: Cannot safely test complex queries or transformations without risking production data. Branching provides instant, isolated testing environments.

• Compliance Officers: Struggle to capture point-in-time snapshots for audits without disrupting production. Branching creates instant, consistent audit trails.

---

HeliosDB Lite Solution

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                   HeliosDB-Lite Application                  │
│           (REPL, Client Library, Embedded Runtime)           │
├─────────────────────────────────────────────────────────────┤
│  Branch Context    │   SQL Parser    │   Query Executor     │
│  (Session State)   │  (USE BRANCH)   │  (Branch-Aware)      │
├─────────────────────────────────────────────────────────────┤
│              Transaction Manager (Branch Routing)             │
│  ┌──────────────────┬───────────────────┬─────────────────┐ │
│  │ Main Transaction │ Branch Transaction│ Merge Engine    │ │
│  └──────────────────┴───────────────────┴─────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                  Branch Manager (COW Logic)                  │
│  ┌──────────────────────────────────────────────────────┐  │
│  │ Branch Metadata  │  Parent Chain  │  Conflict Detector│  │
│  └──────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│            Storage Engine (RocksDB with COW Keys)            │
│  ┌──────────────────────────────────────────────────────┐  │
│  │ Branch-Prefixed Keys:                                 │  │
│  │   main:table_users:row_1                              │  │
│  │   dev:table_users:row_2  (only delta)                 │  │
│  │   audit:table_orders:row_3  (snapshot)                │  │
│  └──────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│       Catalog Storage (Schema Versioned Per Branch)          │
└─────────────────────────────────────────────────────────────┘
         ↓
    Persistent Storage (RocksDB LSM Trees)

---

Concrete Examples with Code, Config & Architecture

Example 1: Zero-Downtime Schema Migration - Production Deployment

Scenario: SaaS application with 100K users needs to add a JSONB preferences column to the users table. Traditional approach requires maintenance window and manual backups. With branching, test the migration on a dev branch, validate data, then merge to production with zero downtime.

Architecture:

Production Environment
    ↓
 HeliosDB-Lite (Embedded in Application Server)
    ↓
Branch Workflow:
  main (production) ──→ dev (schema migration testing)
         ↓                       ↓
    Live traffic           Test queries
         ↓                       ↓
    No downtime            Validate migration
                                 ↓
                           Merge back to main

Configuration (heliosdb.toml):

[database]
path = "/var/lib/heliosdb/production.db"
memory_limit_mb = 1024
enable_wal = true
page_size = 4096

[branching]
enabled = true
# Automatically garbage collect merged branches after 7 days
gc_merged_branches_after_days = 7
# Keep audit branches indefinitely
keep_audit_branches = true

[branching.storage]
# COW storage optimization
dedup_identical_pages = true
compression = "lz4"

[monitoring]
metrics_enabled = true
track_branch_size = true
alert_on_large_divergence_mb = 500

---

Example 2: A/B Testing with Parallel Data Versions - E-Commerce Platform

Scenario: E-commerce platform wants to test two different product recommendation algorithms. Algorithm A uses collaborative filtering (stores user similarity scores), Algorithm B uses content-based filtering (stores product feature vectors). Each algorithm requires different schema and data structures. Traditional approach requires complex application-level routing and dual writes. With branching, create two branches with different schemas and compare results.

Architecture Pattern:

┌─────────────────────────────────────────────────────────┐
│           E-Commerce Application                        │
├─────────────────────────────────────────────────────────┤
│  A/B Test Controller (Routes Users to Branches)         │
├──────────────────────┬──────────────────────────────────┤
│   Branch A:          │   Branch B:                      │
│   algo_a_collab      │   algo_b_content                 │
├──────────────────────┼──────────────────────────────────┤
│ Collaborative Filter │ Content-Based Filter             │
│ Schema:              │ Schema:                          │
│ - user_similarity    │ - product_features               │
│                      │ - user_preferences               │
├──────────────────────┴──────────────────────────────────┤
│          Shared Base Schema (products, users)            │
│                    (Inherited from main)                 │
├─────────────────────────────────────────────────────────┤
│            HeliosDB-Lite Storage Engine                  │
└─────────────────────────────────────────────────────────┘

---

Example 3: Development/Staging/Production Workflow - Microservices Team

Scenario: Team of 5 developers building a microservice that manages customer orders. Each developer needs isolated environment for testing, plus shared staging for integration testing before production. Traditional approach requires 7 separate databases (5 dev + staging + prod). With branching, use single database with 7 branches.

Docker Deployment (Dockerfile):

FROM rust:1.75-slim as builder

WORKDIR /app

# Install HeliosDB-Lite
COPY Cargo.toml Cargo.lock ./
COPY src ./src
RUN cargo build --release

# Runtime stage
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y \
    ca-certificates \
    curl \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/target/release/order-service /usr/local/bin/

# Create data volume for HeliosDB-Lite
RUN mkdir -p /data/heliosdb

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

VOLUME ["/data"]

ENTRYPOINT ["order-service"]
CMD ["--config", "/etc/order-service/config.toml"]

Docker Compose for Multi-Branch Workflow (docker-compose.yml):

version: '3.8'

services:
  # Shared HeliosDB-Lite instance with branch isolation
  order-db:
    image: heliosdb-lite:<version>
    container_name: order-db-shared
    volumes:
      - order_data:/data/heliosdb
      - ./config/heliosdb.toml:/etc/heliosdb/config.toml:ro
    environment:
      HELIOSDB_DATA_DIR: "/data/heliosdb"
      RUST_LOG: "info"
    networks:
      - order-network

  # Production service (uses main branch)
  order-service-prod:
    build: .
    image: order-service:latest
    container_name: order-service-prod
    ports:
      - "8080:8080"
    environment:
      DATABASE_PATH: "/data/heliosdb/orders.db"
      HELIOSDB_BRANCH: "main"  # Production branch
      SERVICE_ENV: "production"
    volumes:
      - order_data:/data/heliosdb:ro  # Read-only in prod
    depends_on:
      - order-db
    networks:
      - order-network

  # Staging service (uses staging branch)
  order-service-staging:
    image: order-service:latest
    container_name: order-service-staging
    ports:
      - "8081:8080"
    environment:
      DATABASE_PATH: "/data/heliosdb/orders.db"
      HELIOSDB_BRANCH: "staging"  # Staging branch
      SERVICE_ENV: "staging"
    volumes:
      - order_data:/data/heliosdb
    depends_on:
      - order-db
    networks:
      - order-network

  # Developer 1 service (uses dev_alice branch)
  order-service-dev-alice:
    image: order-service:latest
    container_name: order-service-dev-alice
    ports:
      - "8082:8080"
    environment:
      DATABASE_PATH: "/data/heliosdb/orders.db"
      HELIOSDB_BRANCH: "dev_alice"  # Alice's dev branch
      SERVICE_ENV: "development"
    volumes:
      - order_data:/data/heliosdb
    depends_on:
      - order-db
    networks:
      - order-network

  # Developer 2 service (uses dev_bob branch)
  order-service-dev-bob:
    image: order-service:latest
    container_name: order-service-dev-bob
    ports:
      - "8083:8080"
    environment:
      DATABASE_PATH: "/data/heliosdb/orders.db"
      HELIOSDB_BRANCH: "dev_bob"  # Bob's dev branch
      SERVICE_ENV: "development"
    volumes:
      - order_data:/data/heliosdb
    depends_on:
      - order-db
    networks:
      - order-network

networks:
  order-network:
    driver: bridge

volumes:
  order_data:
    driver: local

Workflow Commands:

# Start all services (prod, staging, 2 dev instances)
docker-compose up -d

# Alice creates an order on her dev branch
curl -X POST http://localhost:8082/orders \
  -H "Content-Type: application/json" \
  -d '{"customer_id": 123, "total": 99.99}'

# Bob creates an order on his dev branch (isolated from Alice)
curl -X POST http://localhost:8083/orders \
  -H "Content-Type: application/json" \
  -d '{"customer_id": 456, "total": 149.99}'

# Alice merges to staging for integration testing
curl -X POST http://localhost:8082/branch/merge \
  -H "Content-Type: application/json" \
  -d '{"target_branch": "staging"}'

# Staging has Alice's changes for QA review
curl http://localhost:8081/orders

# Production is unaffected until staging merges to main
curl http://localhost:8080/orders

---

Example 4: Audit Trail Snapshots - Financial Compliance

Scenario: Financial services application must capture exact database state at end of each quarter for regulatory audits (SOX, GDPR). Traditional approach uses pg_dump or database backups (slow, large files, difficult to query). With point-in-time branching, create instant audit snapshots that remain queryable.

Configuration (heliosdb.toml):

[database]
path = "/var/lib/heliosdb/financial.db"
memory_limit_mb = 2048
enable_wal = true
wal_checkpoint_interval_secs = 60

[branching]
enabled = true
# Never auto-GC audit branches
gc_merged_branches_after_days = 0
keep_audit_branches = true

[branching.audit]
# Automatically create quarterly snapshots
enable_scheduled_snapshots = true
snapshot_schedule = "0 0 1 1,4,7,10 *"  # First day of Q1, Q2, Q3, Q4
snapshot_prefix = "audit_"
snapshot_naming = "audit_{timestamp}"

[compliance]
# Track who queries audit branches
log_audit_branch_access = true
audit_log_path = "/var/log/heliosdb/audit_access.log"
require_justification = true

CLI Workflow for Auditors:

# Auditor connects to database
$ heliosdb-lite /var/lib/heliosdb/financial.db

heliosdb> \branches
Branch Name         | Created At          | Size      | Description
--------------------|---------------------|-----------|----------------------------
main                | 2024-01-01 00:00:00 | 1.2 GB    | Production database
audit_2024_Q1       | 2024-03-31 23:59:59 | 52 MB     | Quarterly snapshot Q1 2024
audit_2024_Q2       | 2024-06-30 23:59:59 | 78 MB     | Quarterly snapshot Q2 2024
audit_2024_Q3       | 2024-09-30 23:59:59 | 91 MB     | Quarterly snapshot Q3 2024
audit_2024_Q4       | 2024-12-31 23:59:59 | 105 MB    | Quarterly snapshot Q4 2024

heliosdb> USE BRANCH audit_2024_Q4;
Switched to branch: audit_2024_Q4 (read-only)

heliosdb [audit_2024_Q4]> SELECT COUNT(*) FROM accounts WHERE balance > 100000;
+----------+
| COUNT(*) |
+----------+
| 1247     |
+----------+
(1 row)

heliosdb [audit_2024_Q4]> SELECT account_id, balance FROM accounts ORDER BY balance DESC LIMIT 10;
+------------+-----------+
| account_id | balance   |
+------------+-----------+
| 50123      | 5420180.22|
| 50456      | 4892100.50|
...

# Attempt to modify audit branch (will fail - read-only)
heliosdb [audit_2024_Q4]> DELETE FROM accounts WHERE id = 1;
Error: Cannot modify read-only branch: audit_2024_Q4

---

Adoption Strategy

Phase 1: Proof of Concept (Weeks 1-4)

Target: Validate branching for a single high-risk schema migration

Tactics:

• Identify upcoming schema migration (e.g., add column, new index)
• Deploy HeliosDB Lite in dev environment with branching enabled
• Create migration branch, apply schema change, run tests
• Measure: branch creation time, storage overhead, merge success
• Document time savings vs. traditional migration process

Deliverables:

• Migration playbook documenting branching workflow
• Before/after metrics (time, storage, risk)
• Executive summary for stakeholders

Phase 2: Pilot Deployment (Weeks 5-12)

Target: Adopt branching for all schema migrations in one service/team

Tactics:

• Train 1 team (5-10 developers) on branching workflows
• Establish branch naming conventions (e.g., migration_add_user_prefs_20241130)
• Integrate branching into CI/CD pipeline:

   # .github/workflows/schema-migration.yml
   - name: Test migration on branch
     run: |
       heliosdb-cli create-branch migration_${{ github.run_id }} from main
       heliosdb-cli use-branch migration_${{ github.run_id }}
       heliosdb-cli execute schema/migrations/${{ matrix.migration }}.sql
       npm run test:integration
       heliosdb-cli merge-branch migration_${{ github.run_id }} into staging

• Monitor: merge conflicts, rollback frequency, developer satisfaction
• Collect feedback, refine workflows

Deliverables:

• CI/CD templates for branch-based migrations
• Team training materials
• Metrics dashboard (branch usage, merge success rate, storage savings)

Phase 3: Full Rollout (Weeks 13-24)

Target: Organization-wide adoption across all services and teams

Tactics:

• Expand to all microservices (10-50 services)
• Establish governance policies:
• Branch retention: Delete merged branches after 30 days
• Audit branches: Retain quarterly snapshots for 7 years
• Naming conventions: {type}_{description}_{date} (e.g., dev_alice_20241130)
• Automate branch lifecycle:

   [branching.automation]
   # Auto-create audit snapshots
   scheduled_snapshots = "0 0 1 1,4,7,10 *"  # Quarterly

   # Auto-GC merged branches
   gc_merged_after_days = 30

   # Alert on large divergence
   alert_branch_size_threshold_mb = 500

• Integrate with monitoring (Datadog, Prometheus):

   # Track branch metrics
   metrics.gauge('heliosdb.branch.count', len(db.list_branches()))
   metrics.gauge('heliosdb.branch.storage_overhead_pct',
                 get_branch_overhead_percent())

Deliverables:

• Enterprise branching playbook
• Governance policies and automation
• Cost savings report for finance
• Case study for marketing

Phase 4: Advanced Use Cases

Target: Unlock new capabilities enabled by branching

Tactics:

• A/B Testing: Create experiment branches for feature variants
• Tenant Isolation: Per-customer dev branches for white-label SaaS
• Compliance: Automated audit snapshots for SOX, GDPR, HIPAA
• Chaos Engineering: Create failure scenarios on test branches
• Data Science: Isolated experiment branches for ML feature engineering

Example - Chaos Engineering:

# Create chaos branch
heliosdb-cli create-branch chaos_network_partition from prod_snapshot

# Simulate network partition (delete rows)
heliosdb-cli use-branch chaos_network_partition
heliosdb-cli execute "DELETE FROM orders WHERE region = 'us-west' AND created_at > '2024-11-30'"

# Run application tests against chaos branch
npm run test:chaos

# Observe impact, validate recovery procedures

# Delete chaos branch
heliosdb-cli delete-branch chaos_network_partition

---

Conclusion

HeliosDB Lite's Database Branching feature fundamentally transforms how teams manage database change, moving from high-risk, linear migrations to safe, parallel development workflows.

Beyond traditional DevOps use cases, the feature enables entirely new workflows—instant compliance snapshots for auditors, branch-based A/B testing for data scientists, and per-tenant dev environments for SaaS platforms—all without the complexity of external tooling or cloud dependencies.

Call to Action: Start your Database Branching proof of concept today. Identify your next high-risk schema migration, deploy HeliosDB Lite in a dev environment, and experience the power of zero-downtime, zero-risk database evolution. Contact our team for implementation support and enterprise licensing options.