HeliosDB Nano Phase 3 v2.0 Testing Strategy

HeliosDB Nano Phase 3 v2.0 Testing Strategy

Version: 1.0 Created: 2025-11-17 Purpose: Comprehensive testing strategy for Phase 3 features Target Coverage: 90%+ for new features

---

Executive Summary

This document outlines the comprehensive testing strategy for HeliosDB Nano Phase 3 v2.0 features, including:

• SQL Wrapper Layer
• Product Quantization (PQ) for vectors
• Incremental Materialized Views
• FSST/ALP Compression
• Vectorized Execution
• Time-Series Optimizations

Quality Goals:

• ✅ 90%+ code coverage for new Phase 3 features
• ✅ 100% critical path coverage
• ✅ Zero regressions in existing features
• ✅ Performance targets met for all features

---

Testing Pyramid

         /\
        /E2E\          <- 10% (High-value integration scenarios)
       /------\
      /Integr.\        <- 30% (Feature integration tests)
     /----------\
    /   Unit     \     <- 60% (Fast, focused unit tests)
   /--------------\

Distribution:

• Unit Tests (60%): Fast, isolated, comprehensive
• Integration Tests (30%): Feature interactions, end-to-end flows
• E2E Tests (10%): Critical user scenarios

---

Test Categories

1. Unit Tests

1.1 SQL Wrapper Tests

Location: tests/unit/sql_wrapper/

Coverage:

• SQL parser extensions (CREATE MATERIALIZED VIEW, USING hnsw, etc.)
• System view implementations (pg_mv_staleness, pg_vector_index_stats)
• SQL validator logic
• Query rewriting rules

Example Test Structure:

tests/unit/sql_wrapper/parser_tests.rs

#[test]
fn test_parse_create_materialized_view_with_auto_refresh() {
    let sql = r#"
        CREATE MATERIALIZED VIEW user_stats AS
        SELECT user_id, COUNT(*) as order_count
        FROM orders
        GROUP BY user_id
        WITH (auto_refresh = true, max_cpu_percent = 15)
    "#;

    let result = parse_sql(sql);
    assert!(result.is_ok());

    let stmt = result.unwrap();
    assert!(matches!(stmt, Statement::CreateMaterializedView(_)));

    let options = extract_options(&stmt);
    assert_eq!(options.get("auto_refresh"), Some(&"true".to_string()));
    assert_eq!(options.get("max_cpu_percent"), Some(&"15".to_string()));
}

#[test]
fn test_parse_vector_index_with_pq() {
    let sql = r#"
        CREATE INDEX vec_idx ON documents
        USING hnsw (embedding vector_cosine_ops)
        WITH (quantization = 'product', pq_subquantizers = 8, m = 16, ef_construction = 200)
    "#;

    let result = parse_sql(sql);
    assert!(result.is_ok());

    let stmt = result.unwrap();
    let options = extract_index_options(&stmt);
    assert_eq!(options.get("quantization"), Some(&"product".to_string()));
    assert_eq!(options.get("pq_subquantizers"), Some(&"8".to_string()));
}

1.2 Product Quantization Tests

Location: tests/unit/product_quantization/

Coverage:

• K-means clustering algorithm
• Vector encoding/decoding
• Codebook training
• Asymmetric Distance Computation (ADC)
• Sub-quantizer splitting logic
• SIMD optimizations

Example Test Structure:

tests/unit/product_quantization/kmeans_tests.rs

#[test]
fn test_kmeans_clustering_convergence() {
    let vectors = generate_random_vectors(1000, 96); // 1000 vectors, 96 dims
    let k = 256; // 256 centroids

    let kmeans = KMeans::new(k, 100); // 100 iterations
    let (centroids, labels) = kmeans.fit(&vectors);

    assert_eq!(centroids.len(), k);
    assert_eq!(labels.len(), vectors.len());

    // Verify convergence: all vectors assigned to nearest centroid
    for (i, &label) in labels.iter().enumerate() {
        let assigned_centroid = &centroids[label];
        let distance = euclidean_distance(&vectors[i], assigned_centroid);

        // Check all other centroids are farther
        for (j, centroid) in centroids.iter().enumerate() {
            if j != label {
                let other_distance = euclidean_distance(&vectors[i], centroid);
                assert!(distance <= other_distance);
            }
        }
    }
}

#[test]
fn test_product_quantizer_encode_decode() {
    let dim = 768;
    let num_subquantizers = 8;
    let subdim = dim / num_subquantizers; // 96

    let pq = ProductQuantizer::new(dim, num_subquantizers, 256);

    // Train on sample data
    let training_vectors = generate_random_vectors(10000, dim);
    pq.train(&training_vectors).unwrap();

    // Test encoding/decoding
    let original = generate_random_vector(dim);
    let encoded = pq.encode(&original).unwrap();
    let decoded = pq.decode(&encoded).unwrap();

    assert_eq!(encoded.len(), num_subquantizers); // 8 bytes
    assert_eq!(decoded.len(), dim);

    // Quantization error should be reasonable (<10% on average)
    let error = mean_squared_error(&original, &decoded);
    assert!(error < 0.1);
}

#[test]
fn test_asymmetric_distance_computation() {
    let dim = 768;
    let pq = ProductQuantizer::new(dim, 8, 256);

    let training_vectors = generate_random_vectors(10000, dim);
    pq.train(&training_vectors).unwrap();

    let query = generate_random_vector(dim);
    let vector = generate_random_vector(dim);
    let encoded_vector = pq.encode(&vector).unwrap();

    // Compute distance using ADC
    let adc_distance = pq.compute_asymmetric_distance(&query, &encoded_vector);

    // Compute ground truth distance
    let decoded_vector = pq.decode(&encoded_vector).unwrap();
    let true_distance = euclidean_distance(&query, &decoded_vector);

    // ADC should approximate true distance closely (<5% error)
    let relative_error = (adc_distance - true_distance).abs() / true_distance;
    assert!(relative_error < 0.05);
}

1.3 Compression Algorithm Tests

Location: tests/unit/compression/

Coverage:

• FSST codec correctness
• ALP codec correctness
• Compression ratio validation
• Compression/decompression speed
• Edge cases (empty data, single value, etc.)

Example Test Structure:

tests/unit/compression/fsst_tests.rs

#[test]
fn test_fsst_compression_correctness() {
    let data = b"The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy cat.";

    let codec = FsstCodec::new();
    let compressed = codec.compress(data).unwrap();
    let decompressed = codec.decompress(&compressed).unwrap();

    assert_eq!(data, &decompressed[..]);
    assert!(compressed.len() < data.len()); // Should compress
}

#[test]
fn test_fsst_compression_ratio_with_repetitive_text() {
    let repetitive_text = "SELECT * FROM users WHERE user_id = 123; ".repeat(100);
    let data = repetitive_text.as_bytes();

    let codec = FsstCodec::new();
    let compressed = codec.compress(data).unwrap();

    let ratio = data.len() as f64 / compressed.len() as f64;
    assert!(ratio >= 2.0); // At least 2x compression for repetitive text
}

// tests/unit/compression/alp_tests.rs

#[test]
fn test_alp_float_compression() {
    let floats: Vec<f64> = (0..1000)
        .map(|i| 100.0 + (i as f64) * 0.1) // Values with similar exponent
        .collect();

    let codec = AlpCodec::new();
    let compressed = codec.compress(&floats).unwrap();
    let decompressed = codec.decompress(&compressed).unwrap();

    // Verify correctness
    for (i, &original) in floats.iter().enumerate() {
        assert!((original - decompressed[i]).abs() < 1e-10);
    }

    // Verify compression ratio
    let original_bytes = floats.len() * 8;
    let ratio = original_bytes as f64 / compressed.len() as f64;
    assert!(ratio >= 2.0); // At least 2x compression
}

1.4 Incremental MV Tests

Location: tests/unit/incremental_mv/

Coverage:

• Delta tracking logic
• Refresh scheduling
• CPU budget enforcement
• Staleness calculation
• View dependency tracking

2. Integration Tests

2.1 End-to-End SQL Tests

Location: tests/integration/sql_e2e/

Coverage:

• Complete SQL workflows
• Feature interactions
• Transaction handling
• Error scenarios

Example Test Structure:

tests/integration/sql_e2e/materialized_view_integration.rs

#[tokio::test]
async fn test_incremental_mv_auto_refresh_workflow() {
    let db = setup_test_db().await;

    // Create base table
    db.execute("CREATE TABLE orders (user_id INT, amount DECIMAL, created_at TIMESTAMP)")
        .await.unwrap();

    // Create materialized view with auto-refresh
    db.execute(r#"
        CREATE MATERIALIZED VIEW user_totals AS
        SELECT user_id, SUM(amount) as total, COUNT(*) as order_count
        FROM orders
        GROUP BY user_id
        WITH (auto_refresh = true, max_cpu_percent = 15, refresh_interval = '1 second')
    "#).await.unwrap();

    // Insert data
    db.execute("INSERT INTO orders VALUES (1, 100.00, NOW())").await.unwrap();
    db.execute("INSERT INTO orders VALUES (1, 50.00, NOW())").await.unwrap();
    db.execute("INSERT INTO orders VALUES (2, 200.00, NOW())").await.unwrap();

    // Wait for auto-refresh
    tokio::time::sleep(Duration::from_secs(2)).await;

    // Query MV
    let results = db.query("SELECT * FROM user_totals ORDER BY user_id").await.unwrap();

    assert_eq!(results.len(), 2);
    assert_eq!(results[0].get::<i32>("user_id"), 1);
    assert_eq!(results[0].get::<f64>("total"), 150.0);
    assert_eq!(results[0].get::<i64>("order_count"), 2);

    // Check staleness
    let staleness = db.query("SELECT * FROM pg_mv_staleness() WHERE view_name = 'user_totals'")
        .await.unwrap();
    assert!(staleness[0].get::<i64>("staleness_sec") < 5);
}

2.2 Vector Search with PQ Tests

Location: tests/integration/vector_search_pq/

Example Test Structure:

tests/integration/vector_search_pq/pq_search_integration.rs

#[tokio::test]
async fn test_vector_search_with_product_quantization() {
    let db = setup_test_db().await;

    // Create table with vector column
    db.execute("CREATE TABLE documents (id SERIAL PRIMARY KEY, embedding VECTOR(768), content TEXT)")
        .await.unwrap();

    // Create PQ index
    db.execute(r#"
        CREATE INDEX vec_idx ON documents
        USING hnsw (embedding vector_cosine_ops)
        WITH (
            quantization = 'product',
            pq_subquantizers = 8,
            m = 16,
            ef_construction = 200
        )
    "#).await.unwrap();

    // Insert vectors
    let num_vectors = 10000;
    for i in 0..num_vectors {
        let vec = generate_random_vector(768);
        db.execute(&format!(
            "INSERT INTO documents (embedding, content) VALUES ('{}', 'doc{}')",
            vec_to_string(&vec), i
        )).await.unwrap();
    }

    // Search
    let query = generate_random_vector(768);
    let results = db.query(&format!(
        "SELECT id, content FROM documents ORDER BY embedding <=> '{}' LIMIT 10",
        vec_to_string(&query)
    )).await.unwrap();

    assert_eq!(results.len(), 10);

    // Check index stats
    let stats = db.query("SELECT * FROM pg_vector_index_stats('vec_idx')").await.unwrap();
    assert_eq!(stats[0].get::<String>("quantization"), "product");

    let memory_bytes: i64 = stats[0].get("memory_bytes");
    let uncompressed_size = num_vectors * 768 * 4;
    let compression_ratio = uncompressed_size as f64 / memory_bytes as f64;

    // Should have 8-16x compression
    assert!(compression_ratio >= 8.0);
    assert!(compression_ratio <= 20.0);
}

#[tokio::test]
async fn test_pq_search_accuracy() {
    let db = setup_test_db().await;

    // Create table and indexes (both full and PQ)
    db.execute("CREATE TABLE docs (id SERIAL, embedding VECTOR(768))").await.unwrap();

    // Full precision index for ground truth
    db.execute("CREATE INDEX vec_idx_full ON docs USING hnsw (embedding vector_cosine_ops)")
        .await.unwrap();

    // PQ index
    db.execute(r#"
        CREATE INDEX vec_idx_pq ON docs
        USING hnsw (embedding vector_cosine_ops)
        WITH (quantization = 'product', pq_subquantizers = 8)
    "#).await.unwrap();

    // Insert test data
    let num_vectors = 1000;
    for i in 0..num_vectors {
        let vec = generate_random_vector(768);
        db.execute(&format!("INSERT INTO docs (embedding) VALUES ('{}')", vec_to_string(&vec)))
            .await.unwrap();
    }

    // Test multiple queries
    let mut recalls = Vec::new();
    for _ in 0..100 {
        let query = generate_random_vector(768);

        // Ground truth (full precision)
        let ground_truth: Vec<i32> = db.query(&format!(
            "SELECT id FROM docs ORDER BY embedding <=> '{}' LIMIT 10",
            vec_to_string(&query)
        )).await.unwrap()
            .into_iter()
            .map(|row| row.get("id"))
            .collect();

        // PQ results (force using PQ index)
        let pq_results: Vec<i32> = db.query(&format!(
            "SELECT id FROM docs USE INDEX (vec_idx_pq) ORDER BY embedding <=> '{}' LIMIT 10",
            vec_to_string(&query)
        )).await.unwrap()
            .into_iter()
            .map(|row| row.get("id"))
            .collect();

        // Calculate recall@10
        let recall = calculate_recall(&ground_truth, &pq_results);
        recalls.push(recall);
    }

    let avg_recall = recalls.iter().sum::<f64>() / recalls.len() as f64;

    // PQ should maintain 95%+ recall@10
    assert!(avg_recall >= 0.95, "Average recall: {:.2}%", avg_recall * 100.0);
}

3. Performance Benchmarks

Location: benches/phase3_benchmarks.rs

Coverage:

• PQ search speed
• MV refresh speed
• Compression speed
• Decompression speed
• Memory usage

Example Benchmark Structure:

benches/phase3_benchmarks.rs

use criterion::{black_box, criterion_group, criterion_main, Criterion, BenchmarkId};

fn bench_pq_search(c: &mut Criterion) {
    let mut group = c.benchmark_group("pq_search");

    for num_vectors in [10_000, 100_000, 1_000_000] {
        let index = create_pq_index(num_vectors, 768, 8);
        let query = generate_random_vector(768);

        group.bench_with_input(
            BenchmarkId::new("search", num_vectors),
            &num_vectors,
            |b, _| {
                b.iter(|| {
                    let results = index.search(black_box(&query), 10);
                    black_box(results);
                });
            },
        );
    }

    group.finish();
}

fn bench_pq_encoding(c: &mut Criterion) {
    let pq = ProductQuantizer::new(768, 8, 256);
    let training_data = generate_random_vectors(10000, 768);
    pq.train(&training_data).unwrap();

    let vector = generate_random_vector(768);

    c.bench_function("pq_encode", |b| {
        b.iter(|| {
            let encoded = pq.encode(black_box(&vector)).unwrap();
            black_box(encoded);
        });
    });
}

fn bench_fsst_compression(c: &mut Criterion) {
    let mut group = c.benchmark_group("fsst_compression");

    let text_samples = vec![
        ("small", generate_text(1024)),      // 1 KB
        ("medium", generate_text(10240)),    // 10 KB
        ("large", generate_text(102400)),    // 100 KB
    ];

    for (name, text) in text_samples {
        let codec = FsstCodec::new();

        group.bench_with_input(
            BenchmarkId::new("compress", name),
            &text,
            |b, data| {
                b.iter(|| {
                    let compressed = codec.compress(black_box(data)).unwrap();
                    black_box(compressed);
                });
            },
        );

        let compressed = codec.compress(&text).unwrap();
        group.bench_with_input(
            BenchmarkId::new("decompress", name),
            &compressed,
            |b, data| {
                b.iter(|| {
                    let decompressed = codec.decompress(black_box(data)).unwrap();
                    black_box(decompressed);
                });
            },
        );
    }

    group.finish();
}

fn bench_incremental_mv_refresh(c: &mut Criterion) {
    let db = setup_test_db();
    create_mv_with_base_data(&db, 10000); // 10K rows

    c.bench_function("mv_incremental_refresh", |b| {
        b.iter(|| {
            // Insert 10 new rows
            for i in 0..10 {
                db.execute(&format!("INSERT INTO orders VALUES ({}, 100.00)", i));
            }
            // Measure incremental refresh time
            db.refresh_mv("user_totals");
        });
    });
}

criterion_group!(
    benches,
    bench_pq_search,
    bench_pq_encoding,
    bench_fsst_compression,
    bench_incremental_mv_refresh
);
criterion_main!(benches);

4. Compatibility Tests

Location: tests/compatibility/phase3_compatibility.rs

Coverage:

• Backward compatibility with existing features
• Data migration scenarios
• Version upgrade tests
• Feature flag combinations

---

Performance Targets

Product Quantization

• PQ Search (1M vectors): <10ms for k=10
• PQ Encoding: <1ms per vector
• Memory Reduction: 8-16x
• Recall@10: >95%

Incremental MVs

• Incremental Refresh: <100ms for <1000 delta rows
• CPU Usage: <15% when enabled
• Staleness: <5 seconds with auto-refresh

Compression

• FSST Compression Speed: >500 MB/sec
• FSST Decompression Speed: >2 GB/sec
• ALP Compression Speed: >400 MB/sec
• ALP Decompression Speed: >800 MB/sec

---

Test Infrastructure

Test Helpers

Location: tests/test_helpers.rs

Common utilities:

• Database setup/teardown
• Random data generation
• Vector generation
• Assertion helpers
• Performance measurement utilities

CI/CD Integration

GitHub Actions Workflow:

name: Phase 3 Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run unit tests
        run: cargo test --lib --features phase3

  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run integration tests
        run: cargo test --test '*' --features phase3

  benchmarks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run benchmarks
        run: cargo bench --features phase3 -- --save-baseline main

  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Generate coverage report
        run: |
          cargo install cargo-tarpaulin
          cargo tarpaulin --out Xml --features phase3
      - name: Upload coverage
        uses: codecov/codecov-action@v3

---

Test Execution Strategy

Local Development

# Run all unit tests
cargo test --lib --features phase3

# Run integration tests
cargo test --test '*' --features phase3

# Run specific test suite
cargo test --test product_quantization_tests

# Run benchmarks
cargo bench --features phase3

# Generate coverage report
cargo tarpaulin --out Html --features phase3

CI Pipeline

1. PR Checks: Unit + Integration tests (must pass)
2. Nightly: Full test suite + benchmarks
3. Release: Full suite + performance validation

---

Test Metrics and Reporting

Coverage Requirements

• Unit Tests: >90% line coverage
• Integration Tests: 100% critical path coverage
• Overall: >85% coverage for Phase 3 code

Performance Regression Detection

• Benchmark results tracked over time
• Alert on >10% performance regression
• Block merges that fail performance targets

---

Test Maintenance

Best Practices

1. Test Naming: Use descriptive names (test_feature_scenario_expected)
2. Test Isolation: Each test should be independent
3. Test Data: Use factories/builders for test data
4. Assertions: One logical assertion per test
5. Documentation: Comment complex test scenarios

Review Checklist

• All new code has corresponding tests
• Tests cover edge cases
• Performance tests included for critical paths
• Tests are deterministic (no flaky tests)
• Test documentation updated

---

Appendix: Test File Organization

HeliosDB Nano/
├── tests/
│   ├── unit/
│   │   ├── sql_wrapper/
│   │   │   ├── parser_tests.rs
│   │   │   ├── system_views_tests.rs
│   │   │   └── validator_tests.rs
│   │   ├── product_quantization/
│   │   │   ├── kmeans_tests.rs
│   │   │   ├── codec_tests.rs
│   │   │   ├── adc_tests.rs
│   │   │   └── simd_tests.rs
│   │   ├── compression/
│   │   │   ├── fsst_tests.rs
│   │   │   ├── alp_tests.rs
│   │   │   └── ml_selection_tests.rs
│   │   └── incremental_mv/
│   │       ├── delta_tracking_tests.rs
│   │       ├── refresh_tests.rs
│   │       └── cpu_budget_tests.rs
│   ├── integration/
│   │   ├── sql_e2e/
│   │   │   ├── materialized_view_integration.rs
│   │   │   ├── vector_search_integration.rs
│   │   │   └── compression_pipeline_integration.rs
│   │   └── vector_search_pq/
│   │       ├── pq_search_integration.rs
│   │       └── pq_accuracy_tests.rs
│   ├── compatibility/
│   │   └── phase3_compatibility.rs
│   └── test_helpers.rs
├── benches/
│   ├── phase3_benchmarks.rs
│   ├── pq_benchmarks.rs
│   └── compression_benchmarks.rs
└── docs/
    └── testing/
        ├── PHASE3_TEST_STRATEGY.md (this file)
        └── TEST_RESULTS.md (generated)

---

Next Steps

1. Week 1: Implement test infrastructure and helpers
2. Week 2: Write unit tests for SQL Wrapper
3. Week 3: Write unit tests for Product Quantization
4. Week 4: Write integration tests
5. Week 5: Create benchmarks and performance tests
6. Week 6: CI/CD integration and documentation

---

Status: Draft Approval: Pending review Contact: Testing Team Lead