HeliosDB Compatibility Model: Unidirectional (Lite → Full)

HeliosDB Compatibility Model: Unidirectional (Lite → Full)

Version: 1.0 Created: November 15, 2025 Purpose: Clarify the unidirectional compatibility requirement

---

Compatibility Model

Unidirectional: Lite → Full Only

┌─────────────────┐
│  HeliosDB Lite  │
│  (Embedded)     │
└────────┬────────┘
         │
         │  MUST SUPPORT
         │ (Seamless upgrade path)
         │
         ▼
┌─────────────────┐
│ HeliosDB Full   │
│ (Distributed)   │
└─────────────────┘

         ❌ NOT REQUIRED
         (No downgrade path)

What This Means:

• Lite → Full: Must work perfectly (zero data loss, all features preserved)
• ❌ Full → Lite: Not required (can be lossy or not supported at all)

---

Implications for HeliosDB Full

Full Has MORE Freedom, Not Less

Before (Bidirectional assumption):

• ❌ Full constrained to Lite’s feature set
• ❌ Full needs to export to Lite format
• ❌ Full can’t use features Lite doesn’t have

After (Unidirectional reality):

• Full can have features Lite doesn’t have
• Full doesn’t need Lite export capability
• Full can use more sophisticated approaches
• Full only needs to IMPORT from Lite

---

📋 Updated Requirements

For HeliosDB Full

MUST Support (Import from Lite)

1. Import Lite Dumps

   heliosdb-lite export mydb.dump
   heliosdb-full import mydb.dump  #  Must work perfectly

2. Support Lite SQL Syntax (for user familiarity)

   -- Users coming from Lite should recognize syntax
   CREATE DATABASE BRANCH test FROM CURRENT AS OF NOW;
   SELECT * FROM pg_mv_staleness();

3. Preserve Lite Features (in upgraded form)

   • Lite MVs → Full distributed MVs
   • Lite branches → Full distributed branches
   • Lite vector indexes → Full sharded vector indexes

❌ NOT Required (Export to Lite)

1. No Lite Export Format

   heliosdb-full export --lite-format mydb.dump  # ❌ Not needed

2. No Feature Compatibility Checks

   # Full doesn't need to warn if features won't work in Lite
   heliosdb-full check-lite-compatibility  # ❌ Not needed

3. No Downgrade Path

   heliosdb-full downgrade-to-lite  # ❌ Not needed

---

Benefits of Unidirectional Model

1. Full Can Be More Sophisticated

Example: Materialized Views

// Full can use advanced features Lite doesn't have
pub struct FullMaterializedView {
    // Standard features (Lite compatible)
    pub auto_refresh: bool,
    pub max_staleness: Duration,

    // Full-specific (Lite doesn't have these)
    pub ml_optimization: bool,          //  OK - Lite doesn't need this
    pub query_rewriting: bool,          //  OK - Lite doesn't need this
    pub distributed_refresh: bool,      //  OK - Lite is single-node
    pub cross_region_replication: bool, //  OK - Lite doesn't support this
}

Before: “Can we add ML optimization? But Lite doesn’t have it…” After: “Yes! Full can have features Lite doesn’t have.”

2. No Export Complexity

Before (Bidirectional):

// Full would need to maintain two export formats
impl Database {
    pub fn export(&self, format: ExportFormat) -> Result<Dump> {
        match format {
            ExportFormat::Full => self.export_full_format(),
            ExportFormat::Lite => self.export_lite_format(),  // ❌ Complex!
        }
    }

    fn export_lite_format(&self) -> Result<Dump> {
        // Need to strip Full-specific features
        // Need to downgrade distributed data to single-node
        // Need to convert formats
        // LOTS OF WORK!
    }
}

After (Unidirectional):

// Full only needs one export format
impl Database {
    pub fn export(&self) -> Result<Dump> {
        self.export_full_format()  //  Simple!
    }
}

3. Simplified Testing

Before (Bidirectional):

• Test Lite → Full migration
• Test Full → Lite migration (complex!)
• Test round-trip (Lite → Full → Lite)

After (Unidirectional):

• Test Lite → Full migration
• Done!

4. Better User Experience

Typical User Journey:

1. Start with Lite (embedded, development)
2. Grow to Full (production, scale)
3. Stay on Full (never go back)

Edge Case (downgrade):

User: "I want to downgrade from Full to Lite"
Answer: "Full is for production scale. If you need embedded again,
         export your data and reimport to a new Lite instance.
         Note: Some Full features won't be available."

This is acceptable because downgrade is rare and can be manual.

---

Feature Matrix (Updated)

Features: Lite vs Full

Feature	Lite	Full	Migration Lite→Full	Migration Full→Lite
Materialized Views	Basic incremental	ML + distributed	Upgrade to distributed	❌ Not needed
Branching	Local COW	Distributed COW	Distribute	❌ Not needed
Vector Search	Local HNSW	Sharded HNSW	Shard	❌ Not needed
Compression	Auto threshold	ML-based	Use ML	❌ Not needed
CPU Throttling	Mandatory <15%	Optional	Convert to optional	❌ Not needed
Hybrid Storage	Hot/cold tiers	HCC v2	Convert to HCC v2	❌ Not needed
ML Optimization	❌ No	Yes	N/A (Full adds it)	❌ Not needed
Cross-region	❌ No	Yes	N/A (Full adds it)	❌ Not needed
Distributed Query	❌ No	Yes	N/A (Full adds it)	❌ Not needed

Key Insight: Full can have features Lite doesn’t have!

---

Updated Implementation Guidelines

For SQL Compatibility

Requirement: Lite SQL syntax should work in Full (for user familiarity)

NOT a Requirement: Full SQL syntax must work in Lite

Example:

--  Lite syntax (must work in Full)
CREATE MATERIALIZED VIEW user_stats AS
SELECT user_id, COUNT(*) FROM orders GROUP BY user_id
WITH (
    auto_refresh = true,
    max_cpu_percent = 15
);

--  Full syntax (Lite doesn't need to support this)
CREATE MATERIALIZED VIEW user_stats AS
SELECT user_id, COUNT(*) FROM orders GROUP BY user_id
WITH (
    auto_refresh = true,
    ml_optimization = true,           -- Full-specific
    distributed_refresh = true,       -- Full-specific
    cross_region_replication = true,  -- Full-specific
    replication_factor = 3            -- Full-specific
);

Result: Users migrating from Lite to Full can use familiar syntax, AND use new Full features.

---

For Migration Testing

Required Tests:

#[test]
fn test_lite_to_full_migration() {
    //  Must pass
    let lite_dump = export_from_lite();
    let full_db = import_to_full(lite_dump);
    assert_data_integrity(full_db);
    assert_features_upgraded(full_db);
}

NOT Required Tests:

#[test]
fn test_full_to_lite_migration() {
    // ❌ Not needed (can skip this test entirely)
}

#[test]
fn test_round_trip_migration() {
    // ❌ Not needed (can skip this test entirely)
}

---

For Documentation

Required Documentation:

• How to migrate from Lite to Full
• What happens to Lite features in Full
• What new Full features are available

NOT Required Documentation:

• ❌ How to migrate from Full to Lite
• ❌ Feature parity between Lite and Full
• ❌ Downgrade procedures

---

Revised Compatibility Statement

Official Compatibility Guarantee

HeliosDB Lite is designed as an embedded database with a seamless
upgrade path to HeliosDB Full.

 Guaranteed:
- All Lite data can be migrated to Full without loss
- All Lite features are preserved (and enhanced) in Full
- Lite SQL syntax works in Full

❌ Not Guaranteed:
- Full data cannot be migrated back to Lite
- Full features are not available in Lite
- Full SQL syntax may not work in Lite

Recommendation: Start with Lite for development/embedded use.
                Upgrade to Full for production/scale.
                Do not downgrade from Full to Lite.

---

🎉 Simplified Implementation Plan

What This Changes

❌ REMOVED from Requirements

1. Full → Lite export format (not needed)
2. Feature compatibility checks (not needed)
3. Downgrade testing (not needed)
4. Round-trip testing (not needed)
5. Lite feature constraints on Full (not needed)

SIMPLIFIED Requirements

1. Import from Lite (already planned)
2. Support Lite SQL syntax (already planned)
3. Upgrade Lite features (already planned)

Result: Implementation is EASIER than before!

---

Implementation Remains Unchanged

Good News: Already Designed Correctly!

All documents already assume Lite → Full migration:

• PHASE3_FULL_IMPLEMENTATION_GUIDE.md
• SQL_WRAPPER_SPECIFICATION.md
• PRODUCT_QUANTIZATION_IMPLEMENTATION.md
• INCREMENTAL_MVS_DISTRIBUTED.md

What Changes: Mindset

Old Mindset:

• “Full must be compatible with Lite” (constraining)

New Mindset:

• “Full must accept Lite data” (liberating)
• “Full can have features Lite doesn’t have” (empowering)

---

Updated Timeline (Unchanged)

P0: Critical (5-7 weeks)

1. SQL Wrapper (2-3 weeks) - Support Lite SQL syntax
2. Product Quantization (3-4 weeks) - Works in both

P1: High (6-8 weeks)

3. Distributed Incremental MVs (4-5 weeks) - Upgrade Lite MVs
4. FSST + ALP Compression (2-3 weeks) - Enhance compression

No changes needed - implementation plan already correct!

---

Final Checklist

For HeliosDB Full Team

• Understand compatibility is unidirectional (Lite → Full)
• Full can have features Lite doesn’t have
• Full doesn’t need Lite export format
• Focus on importing from Lite, not exporting to Lite
• Implementation plan already correct
• No additional work required

For HeliosDB Lite Team

• Ensure export format includes all Phase 3 features
• Document what gets upgraded during Full migration
• Test Lite → Full migration thoroughly
• No need to worry about Full → Lite

---

Summary

Unidirectional compatibility (Lite → Full) means:

Full MUST:

• Import Lite dumps perfectly
• Support Lite SQL syntax
• Preserve and enhance Lite features

Full CAN:

• Have features Lite doesn’t have
• Use more sophisticated approaches
• Ignore Lite’s limitations

❌ Full DOES NOT NEED TO:

• Export to Lite format
• Constrain features to Lite’s capabilities
• Support downgrade from Full to Lite

Result: Implementation is simpler and Full is more powerful!

---

Status: Clarified Impact: Positive (simplifies implementation) Changes to Plan: None needed (already designed correctly)