HeliosDB Compatibility Model: Unidirectional (Lite → Full)

HeliosDB Compatibility Model: Unidirectional (Lite → Full)

Version: 1.0 Created: November 15, 2025 Purpose: Clarify the unidirectional compatibility requirement

---

✅ Compatibility Model

Unidirectional: Lite → Full Only

┌─────────────────┐
│  HeliosDB Nano  │
│  (Embedded)     │
└────────┬────────┘
         │
         │ ✅ MUST SUPPORT
         │ (Seamless upgrade path)
         │
         ▼
┌─────────────────┐
│ HeliosDB Full   │
│ (Distributed)   │
└─────────────────┘

         ❌ NOT REQUIRED
         (No downgrade path)

What This Means:

• ✅ Lite → Full: Must work perfectly (zero data loss, all features preserved)
• ❌ Full → Lite: Not required (can be lossy or not supported at all)

---

🎯 Implications for HeliosDB Full

Full Has MORE Freedom, Not Less

Before (Bidirectional assumption):

• ❌ Full constrained to Lite’s feature set
• ❌ Full needs to export to Lite format
• ❌ Full can’t use features Lite doesn’t have

After (Unidirectional reality):

• ✅ Full can have features Lite doesn’t have
• ✅ Full doesn’t need Lite export capability
• ✅ Full can use more sophisticated approaches
• ✅ Full only needs to IMPORT from Lite

---

📋 Updated Requirements

For HeliosDB Full

✅ MUST Support (Import from Lite)

1. Import Lite Dumps

   heliosdb-nano export mydb.dump
   heliosdb-full import mydb.dump  # ✅ Must work perfectly

2. Support Lite SQL Syntax (for user familiarity)

   -- Users coming from Lite should recognize syntax
   CREATE DATABASE BRANCH test FROM CURRENT AS OF NOW;
   SELECT * FROM pg_mv_staleness();

3. Preserve Lite Features (in upgraded form)

   • Lite MVs → Full distributed MVs
   • Lite branches → Full distributed branches
   • Lite vector indexes → Full sharded vector indexes

❌ NOT Required (Export to Lite)

1. No Lite Export Format

   heliosdb-full export --lite-format mydb.dump  # ❌ Not needed

2. No Feature Compatibility Checks

   # Full doesn't need to warn if features won't work in Lite
   heliosdb-full check-lite-compatibility  # ❌ Not needed

3. No Downgrade Path

   heliosdb-full downgrade-to-lite  # ❌ Not needed

---

🚀 Benefits of Unidirectional Model

1. Full Can Be More Sophisticated

Example: Materialized Views

// Full can use advanced features Lite doesn't have
pub struct FullMaterializedView {
    // Standard features (Lite compatible)
    pub auto_refresh: bool,
    pub max_staleness: Duration,

    // Full-specific (Lite doesn't have these)
    pub ml_optimization: bool,          // ✅ OK - Lite doesn't need this
    pub query_rewriting: bool,          // ✅ OK - Lite doesn't need this
    pub distributed_refresh: bool,      // ✅ OK - Lite is single-node
    pub cross_region_replication: bool, // ✅ OK - Lite doesn't support this
}

Before: “Can we add ML optimization? But Lite doesn’t have it…” After: “Yes! Full can have features Lite doesn’t have.”

2. No Export Complexity

Before (Bidirectional):

// Full would need to maintain two export formats
impl Database {
    pub fn export(&self, format: ExportFormat) -> Result<Dump> {
        match format {
            ExportFormat::Full => self.export_full_format(),
            ExportFormat::Lite => self.export_lite_format(),  // ❌ Complex!
        }
    }

    fn export_lite_format(&self) -> Result<Dump> {
        // Need to strip Full-specific features
        // Need to downgrade distributed data to single-node
        // Need to convert formats
        // LOTS OF WORK!
    }
}

After (Unidirectional):

// Full only needs one export format
impl Database {
    pub fn export(&self) -> Result<Dump> {
        self.export_full_format()  // ✅ Simple!
    }
}

3. Simplified Testing

Before (Bidirectional):

• Test Lite → Full migration
• Test Full → Lite migration (complex!)
• Test round-trip (Lite → Full → Lite)

After (Unidirectional):

• Test Lite → Full migration ✅
• Done!

4. Better User Experience

Typical User Journey:

1. Start with Lite (embedded, development)
2. Grow to Full (production, scale)
3. Stay on Full (never go back)

Edge Case (downgrade):

User: "I want to downgrade from Full to Lite"
Answer: "Full is for production scale. If you need embedded again,
         export your data and reimport to a new Lite instance.
         Note: Some Full features won't be available."

This is acceptable because downgrade is rare and can be manual.

---

📊 Feature Matrix (Updated)

Features: Lite vs Full

Feature	Lite	Full	Migration Lite→Full	Migration Full→Lite
Materialized Views	Basic incremental	ML + distributed	✅ Upgrade to distributed	❌ Not needed
Branching	Local COW	Distributed COW	✅ Distribute	❌ Not needed
Vector Search	Local HNSW	Sharded HNSW	✅ Shard	❌ Not needed
Compression	Auto threshold	ML-based	✅ Use ML	❌ Not needed
CPU Throttling	Mandatory <15%	Optional	✅ Convert to optional	❌ Not needed
Hybrid Storage	Hot/cold tiers	HCC v2	✅ Convert to HCC v2	❌ Not needed
ML Optimization	❌ No	✅ Yes	N/A (Full adds it)	❌ Not needed
Cross-region	❌ No	✅ Yes	N/A (Full adds it)	❌ Not needed
Distributed Query	❌ No	✅ Yes	N/A (Full adds it)	❌ Not needed

Key Insight: Full can have features Lite doesn’t have!

---

🎯 Updated Implementation Guidelines

For SQL Compatibility

Requirement: Lite SQL syntax should work in Full (for user familiarity)

NOT a Requirement: Full SQL syntax must work in Lite

Example:

-- ✅ Lite syntax (must work in Full)
CREATE MATERIALIZED VIEW user_stats AS
SELECT user_id, COUNT(*) FROM orders GROUP BY user_id
WITH (
    auto_refresh = true,
    max_cpu_percent = 15
);

-- ✅ Full syntax (Lite doesn't need to support this)
CREATE MATERIALIZED VIEW user_stats AS
SELECT user_id, COUNT(*) FROM orders GROUP BY user_id
WITH (
    auto_refresh = true,
    ml_optimization = true,           -- Full-specific
    distributed_refresh = true,       -- Full-specific
    cross_region_replication = true,  -- Full-specific
    replication_factor = 3            -- Full-specific
);

Result: Users migrating from Lite to Full can use familiar syntax, AND use new Full features.

---

For Migration Testing

Required Tests:

#[test]
fn test_lite_to_full_migration() {
    // ✅ Must pass
    let lite_dump = export_from_lite();
    let full_db = import_to_full(lite_dump);
    assert_data_integrity(full_db);
    assert_features_upgraded(full_db);
}

NOT Required Tests:

#[test]
fn test_full_to_lite_migration() {
    // ❌ Not needed (can skip this test entirely)
}

#[test]
fn test_round_trip_migration() {
    // ❌ Not needed (can skip this test entirely)
}

---

For Documentation

Required Documentation:

• ✅ How to migrate from Lite to Full
• ✅ What happens to Lite features in Full
• ✅ What new Full features are available

NOT Required Documentation:

• ❌ How to migrate from Full to Lite
• ❌ Feature parity between Lite and Full
• ❌ Downgrade procedures

---

📝 Revised Compatibility Statement

Official Compatibility Guarantee

HeliosDB Nano is designed as an embedded database with a seamless
upgrade path to HeliosDB Full.

✅ Guaranteed:
- All Lite data can be migrated to Full without loss
- All Lite features are preserved (and enhanced) in Full
- Lite SQL syntax works in Full

❌ Not Guaranteed:
- Full data cannot be migrated back to Lite
- Full features are not available in Lite
- Full SQL syntax may not work in Lite

Recommendation: Start with Lite for development/embedded use.
                Upgrade to Full for production/scale.
                Do not downgrade from Full to Lite.

---

🎉 Simplified Implementation Plan

What This Changes

❌ REMOVED from Requirements

1. Full → Lite export format (not needed)
2. Feature compatibility checks (not needed)
3. Downgrade testing (not needed)
4. Round-trip testing (not needed)
5. Lite feature constraints on Full (not needed)

✅ SIMPLIFIED Requirements

1. Import from Lite (already planned)
2. Support Lite SQL syntax (already planned)
3. Upgrade Lite features (already planned)

Result: Implementation is EASIER than before!

---

🚀 Implementation Remains Unchanged

Good News: Already Designed Correctly!

All documents already assume Lite → Full migration:

• ✅ PHASE3_FULL_IMPLEMENTATION_GUIDE.md
• ✅ SQL_WRAPPER_SPECIFICATION.md
• ✅ PRODUCT_QUANTIZATION_IMPLEMENTATION.md
• ✅ INCREMENTAL_MVS_DISTRIBUTED.md

What Changes: Mindset

Old Mindset:

• “Full must be compatible with Lite” (constraining)

New Mindset:

• “Full must accept Lite data” (liberating)
• “Full can have features Lite doesn’t have” (empowering)

---

📊 Updated Timeline (Unchanged)

P0: Critical (5-7 weeks)

1. SQL Wrapper (2-3 weeks) - Support Lite SQL syntax
2. Product Quantization (3-4 weeks) - Works in both

P1: High (6-8 weeks)

3. Distributed Incremental MVs (4-5 weeks) - Upgrade Lite MVs
4. FSST + ALP Compression (2-3 weeks) - Enhance compression

No changes needed - implementation plan already correct!

---

✅ Final Checklist

For HeliosDB Full Team

• Understand compatibility is unidirectional (Lite → Full)
• Full can have features Lite doesn’t have
• Full doesn’t need Lite export format
• Focus on importing from Lite, not exporting to Lite
• Implementation plan already correct
• No additional work required

For HeliosDB Nano Team

• Ensure export format includes all Phase 3 features
• Document what gets upgraded during Full migration
• Test Lite → Full migration thoroughly
• No need to worry about Full → Lite

---

🎯 Summary

Unidirectional compatibility (Lite → Full) means:

✅ Full MUST:

• Import Lite dumps perfectly
• Support Lite SQL syntax
• Preserve and enhance Lite features

✅ Full CAN:

• Have features Lite doesn’t have
• Use more sophisticated approaches
• Ignore Lite’s limitations

❌ Full DOES NOT NEED TO:

• Export to Lite format
• Constrain features to Lite’s capabilities
• Support downgrade from Full to Lite

Result: Implementation is simpler and Full is more powerful!

---

Status: ✅ Clarified Impact: ✅ Positive (simplifies implementation) Changes to Plan: ✅ None needed (already designed correctly)