Group Commit WAL Test Strategy

Group Commit WAL Test Strategy

Document Version: 1.0 Date: 2025-11-10 Status: Test Plan Related: GROUP_COMMIT_WAL_ARCHITECTURE.md

Overview

This document defines the comprehensive testing strategy for the Group Commit WAL system, ensuring correctness, performance, and production readiness.

---

Test Pyramid

                    ┌─────────────┐
                    │   Chaos     │ (5%)
                    │   Tests     │
                    └─────────────┘
                  ┌─────────────────┐
                  │   Performance   │ (10%)
                  │   Tests         │
                  └─────────────────┘
              ┌───────────────────────┐
              │   Integration Tests   │ (25%)
              └───────────────────────┘
          ┌─────────────────────────────┐
          │       Unit Tests            │ (60%)
          └─────────────────────────────┘

Test Distribution:

• Unit Tests: 60% - Fast, focused, high coverage
• Integration Tests: 25% - System interactions
• Performance Tests: 10% - Benchmarks, regression
• Chaos Tests: 5% - Failure injection, resilience

---

1. Unit Tests (Target: 80% code coverage)

1.1 LSN Management Tests

File: heliosdb-storage/src/wal/group_commit/lsn_counter_tests.rs

#[cfg(test)]
mod lsn_counter_tests {
    use super::*;

    #[test]
    fn test_lsn_initial_value() {
        let counter = LsnCounter::new(100);
        assert_eq!(counter.current(), Lsn::new(100));
    }

    #[test]
    fn test_lsn_monotonicity() {
        let counter = LsnCounter::new(0);
        let lsn1 = counter.next();
        let lsn2 = counter.next();
        let lsn3 = counter.next();

        assert!(lsn1 < lsn2);
        assert!(lsn2 < lsn3);
        assert_eq!(lsn1.value() + 1, lsn2.value());
    }

    #[test]
    fn test_lsn_concurrent_assignment() {
        let counter = Arc::new(LsnCounter::new(0));
        let mut handles = vec![];

        for _ in 0..10 {
            let c = Arc::clone(&counter);
            handles.push(thread::spawn(move || {
                (0..1000).map(|_| c.next()).collect::<Vec<_>>()
            }));
        }

        let all_lsns: Vec<Lsn> = handles
            .into_iter()
            .flat_map(|h| h.join().unwrap())
            .collect();

        // All LSNs should be unique
        let unique: HashSet<_> = all_lsns.iter().cloned().collect();
        assert_eq!(unique.len(), 10_000);

        // Should be consecutive from 1 to 10000
        let mut sorted = all_lsns;
        sorted.sort();
        assert_eq!(sorted.first().unwrap().value(), 1);
        assert_eq!(sorted.last().unwrap().value(), 10_000);
    }

    #[test]
    fn test_lsn_ordering() {
        let lsn1 = Lsn::new(1);
        let lsn2 = Lsn::new(2);
        let lsn3 = Lsn::new(1); // Duplicate

        assert!(lsn1 < lsn2);
        assert!(lsn1 == lsn3);
        assert!(lsn2 > lsn1);
    }
}

1.2 Pending Queue Tests

File: heliosdb-storage/src/wal/group_commit/pending_queue_tests.rs

#[cfg(test)]
mod pending_queue_tests {
    use super::*;

    #[test]
    fn test_queue_basic_operations() {
        let queue = PendingQueue::new();

        assert_eq!(queue.len(), 0);
        assert!(queue.pop().is_none());

        queue.push(create_test_entry(1));
        assert_eq!(queue.len(), 1);

        let entry = queue.pop().unwrap();
        assert_eq!(entry.lsn, Lsn::new(1));
        assert_eq!(queue.len(), 0);
    }

    #[test]
    fn test_queue_fifo_ordering() {
        let queue = PendingQueue::new();

        for i in 1..=10 {
            queue.push(create_test_entry(i));
        }

        for i in 1..=10 {
            let entry = queue.pop().unwrap();
            assert_eq!(entry.lsn.value(), i);
        }
    }

    #[test]
    fn test_queue_concurrent_push() {
        let queue = Arc::new(PendingQueue::new());
        let mut handles = vec![];

        for i in 0..10 {
            let q = Arc::clone(&queue);
            handles.push(thread::spawn(move || {
                for j in 0..100 {
                    q.push(create_test_entry(i * 100 + j));
                }
            }));
        }

        for h in handles {
            h.join().unwrap();
        }

        assert_eq!(queue.len(), 1000);
    }

    #[test]
    fn test_queue_concurrent_push_pop() {
        let queue = Arc::new(PendingQueue::new());
        let counter = Arc::new(AtomicUsize::new(0));

        // Producers
        let producers: Vec<_> = (0..5)
            .map(|_| {
                let q = Arc::clone(&queue);
                thread::spawn(move || {
                    for i in 0..200 {
                        q.push(create_test_entry(i));
                        thread::sleep(Duration::from_micros(10));
                    }
                })
            })
            .collect();

        // Consumers
        let consumers: Vec<_> = (0..5)
            .map(|_| {
                let q = Arc::clone(&queue);
                let c = Arc::clone(&counter);
                thread::spawn(move || {
                    loop {
                        if let Some(_) = q.pop() {
                            c.fetch_add(1, Ordering::SeqCst);
                        }

                        if c.load(Ordering::SeqCst) >= 1000 {
                            break;
                        }

                        thread::sleep(Duration::from_micros(10));
                    }
                })
            })
            .collect();

        for p in producers {
            p.join().unwrap();
        }

        for c in consumers {
            c.join().unwrap();
        }

        assert_eq!(counter.load(Ordering::SeqCst), 1000);
    }
}

1.3 Batching Logic Tests

File: heliosdb-storage/src/wal/group_commit/batching_tests.rs

#[cfg(test)]
mod batching_tests {
    use super::*;

    #[test]
    fn test_time_based_flush() {
        let config = GroupCommitConfig {
            max_flush_interval_ms: 10,
            max_batch_size: 1000,
            ..Default::default()
        };

        let queue = Arc::new(PendingQueue::new());
        let collector = BatchCollector::new(config, Arc::clone(&queue));

        // Add 10 entries
        for i in 0..10 {
            queue.push(create_test_entry(i));
        }

        // Should flush after 10ms
        let start = Instant::now();
        let batch = collector.collect_batch();
        let elapsed = start.elapsed();

        assert_eq!(batch.len(), 10);
        assert!(elapsed >= Duration::from_millis(9)); // Allow 1ms tolerance
        assert!(elapsed < Duration::from_millis(20));
    }

    #[test]
    fn test_size_based_flush() {
        let config = GroupCommitConfig {
            max_flush_interval_ms: 1000,
            max_batch_size: 50,
            ..Default::default()
        };

        let queue = Arc::new(PendingQueue::new());
        let collector = BatchCollector::new(config, Arc::clone(&queue));

        // Add 50 entries rapidly
        for i in 0..50 {
            queue.push(create_test_entry(i));
        }

        // Should flush immediately when batch reaches 50
        let start = Instant::now();
        let batch = collector.collect_batch();
        let elapsed = start.elapsed();

        assert_eq!(batch.len(), 50);
        assert!(elapsed < Duration::from_millis(100));
    }

    #[test]
    fn test_hybrid_flush_time_wins() {
        let config = GroupCommitConfig {
            max_flush_interval_ms: 10,
            max_batch_size: 100,
            ..Default::default()
        };

        let queue = Arc::new(PendingQueue::new());
        let collector = BatchCollector::new(config, Arc::clone(&queue));

        // Add only 20 entries
        for i in 0..20 {
            queue.push(create_test_entry(i));
        }

        // Should flush after 10ms (time limit)
        let batch = collector.collect_batch();
        assert_eq!(batch.len(), 20);
    }

    #[test]
    fn test_hybrid_flush_size_wins() {
        let config = GroupCommitConfig {
            max_flush_interval_ms: 1000,
            max_batch_size: 100,
            ..Default::default()
        };

        let queue = Arc::new(PendingQueue::new());
        let collector = BatchCollector::new(config, Arc::clone(&queue));

        // Add 100 entries rapidly
        for i in 0..100 {
            queue.push(create_test_entry(i));
        }

        // Should flush immediately (size limit)
        let start = Instant::now();
        let batch = collector.collect_batch();
        let elapsed = start.elapsed();

        assert_eq!(batch.len(), 100);
        assert!(elapsed < Duration::from_millis(100));
    }

    #[test]
    fn test_empty_queue_timeout() {
        let config = GroupCommitConfig {
            max_flush_interval_ms: 10,
            max_batch_size: 100,
            ..Default::default()
        };

        let queue = Arc::new(PendingQueue::new());
        let collector = BatchCollector::new(config, Arc::clone(&queue));

        // No entries - should return empty batch after timeout
        let start = Instant::now();
        let batch = collector.collect_batch();
        let elapsed = start.elapsed();

        assert_eq!(batch.len(), 0);
        assert!(elapsed >= Duration::from_millis(9));
    }
}

1.4 WAL Format Tests

File: heliosdb-storage/src/wal/group_commit/format_tests.rs

#[cfg(test)]
mod format_tests {
    use super::*;

    #[test]
    fn test_encode_decode_roundtrip() {
        let lsn = Lsn::new(42);
        let entry = WalEntry {
            txn_id: 123,
            entry_type: WalEntryType::Commit,
            data: vec![1, 2, 3, 4, 5],
            timestamp: 1234567890,
            checksum: 0,
        };

        let encoded = WalEncoder::encode(lsn, &entry);
        let mut cursor = Cursor::new(&encoded);
        let (decoded_lsn, decoded_entry) = WalEncoder::decode(&mut cursor).unwrap();

        assert_eq!(decoded_lsn, lsn);
        assert_eq!(decoded_entry.txn_id, entry.txn_id);
        assert_eq!(decoded_entry.entry_type, entry.entry_type);
        assert_eq!(decoded_entry.data, entry.data);
    }

    #[test]
    fn test_checksum_validation() {
        let lsn = Lsn::new(1);
        let entry = create_test_entry(1);

        let mut encoded = WalEncoder::encode(lsn, &entry);

        // Corrupt the data
        encoded[20] ^= 0xFF;

        let mut cursor = Cursor::new(&encoded);
        let result = WalEncoder::decode(&mut cursor);

        assert!(result.is_err());
        assert_eq!(result.unwrap_err().kind(), io::ErrorKind::InvalidData);
    }

    #[test]
    fn test_partial_entry_detection() {
        let lsn = Lsn::new(1);
        let entry = create_test_entry(1);

        let encoded = WalEncoder::encode(lsn, &entry);

        // Truncate the encoded data
        let partial = &encoded[..encoded.len() - 10];

        let mut cursor = Cursor::new(partial);
        let result = WalEncoder::decode(&mut cursor);

        assert!(result.is_err());
        assert_eq!(result.unwrap_err().kind(), io::ErrorKind::UnexpectedEof);
    }

    #[test]
    fn test_all_entry_types() {
        for entry_type in [
            WalEntryType::Begin,
            WalEntryType::Data,
            WalEntryType::Commit,
            WalEntryType::Abort,
            WalEntryType::Checkpoint,
        ] {
            let lsn = Lsn::new(1);
            let entry = WalEntry {
                txn_id: 1,
                entry_type,
                data: vec![1, 2, 3],
                timestamp: 1234567890,
                checksum: 0,
            };

            let encoded = WalEncoder::encode(lsn, &entry);
            let mut cursor = Cursor::new(&encoded);
            let (_, decoded) = WalEncoder::decode(&mut cursor).unwrap();

            assert_eq!(decoded.entry_type, entry_type);
        }
    }
}

1.5 Durability Modes Tests

File: heliosdb-storage/src/wal/group_commit/durability_tests.rs

#[cfg(test)]
mod durability_tests {
    use super::*;

    #[tokio::test]
    async fn test_synchronous_mode() {
        let wal = GroupCommitWal::new(
            "/tmp/test_sync",
            GroupCommitConfig {
                durability_mode: DurabilityMode::Synchronous,
                ..Default::default()
            },
        ).unwrap();

        let start = Instant::now();
        let lsn = wal.append_with_mode(
            create_test_entry(1),
            DurabilityMode::Synchronous,
        ).await.unwrap();

        // Should wait for flush
        assert!(start.elapsed() >= Duration::from_millis(5));

        // Should be immediately durable
        assert!(wal.last_flushed_lsn() >= lsn);
    }

    #[tokio::test]
    async fn test_group_commit_mode() {
        let wal = GroupCommitWal::new(
            "/tmp/test_group",
            GroupCommitConfig {
                durability_mode: DurabilityMode::GroupCommit,
                ..Default::default()
            },
        ).unwrap();

        let lsn = wal.append(create_test_entry(1)).unwrap();

        // Should return immediately (not durable yet)
        assert!(wal.last_flushed_lsn() < lsn);

        // Explicitly wait for durability
        wal.wait_for_lsn(lsn).await.unwrap();

        // Now should be durable
        assert!(wal.last_flushed_lsn() >= lsn);
    }

    #[tokio::test]
    async fn test_async_mode() {
        let wal = GroupCommitWal::new(
            "/tmp/test_async",
            GroupCommitConfig {
                durability_mode: DurabilityMode::Async,
                ..Default::default()
            },
        ).unwrap();

        let start = Instant::now();
        let lsn = wal.append_with_mode(
            create_test_entry(1),
            DurabilityMode::Async,
        ).await.unwrap();

        // Should return immediately
        assert!(start.elapsed() < Duration::from_millis(1));

        // May not be durable yet (acceptable for async mode)
        // Don't assert on durability
    }
}

---

2. Integration Tests (Target: All critical paths covered)

2.1 Concurrent Commit Tests

File: heliosdb-storage/tests/group_commit_integration_tests.rs

#[tokio::test]
async fn test_concurrent_commits() {
    let wal = Arc::new(GroupCommitWal::new(
        "/tmp/test_concurrent",
        Default::default(),
    ).unwrap());

    let mut handles = vec![];

    // Spawn 100 concurrent writers
    for i in 0..100 {
        let wal = Arc::clone(&wal);
        handles.push(tokio::spawn(async move {
            for j in 0..100 {
                let entry = create_test_entry(i * 100 + j);
                let lsn = wal.append(entry).unwrap();
                wal.wait_for_lsn(lsn).await.unwrap();
            }
        }));
    }

    // Wait for all
    for h in handles {
        h.await.unwrap();
    }

    // Verify metrics
    let metrics = wal.metrics();
    assert_eq!(metrics.total_appends, 10_000);
    assert!(metrics.total_flushes < 10_000); // Should have batching
}

#[tokio::test]
async fn test_mixed_durability_modes() {
    let wal = Arc::new(GroupCommitWal::new(
        "/tmp/test_mixed",
        Default::default(),
    ).unwrap());

    let mut handles = vec![];

    // Synchronous writers
    for i in 0..10 {
        let wal = Arc::clone(&wal);
        handles.push(tokio::spawn(async move {
            for j in 0..10 {
                wal.append_with_mode(
                    create_test_entry(i * 10 + j),
                    DurabilityMode::Synchronous,
                ).await.unwrap();
            }
        }));
    }

    // Async writers
    for i in 10..20 {
        let wal = Arc::clone(&wal);
        handles.push(tokio::spawn(async move {
            for j in 0..10 {
                wal.append_with_mode(
                    create_test_entry(i * 10 + j),
                    DurabilityMode::Async,
                ).await.unwrap();
            }
        }));
    }

    for h in handles {
        h.await.unwrap();
    }

    assert_eq!(wal.metrics().total_appends, 200);
}

2.2 Recovery Tests

File: heliosdb-storage/tests/recovery_integration_tests.rs

#[test]
fn test_recovery_after_clean_shutdown() {
    let path = "/tmp/test_recovery_clean";

    // Write entries
    {
        let wal = GroupCommitWal::new(path, Default::default()).unwrap();
        for i in 0..100 {
            let lsn = wal.append(create_test_entry(i)).unwrap();
            futures::executor::block_on(wal.wait_for_lsn(lsn)).unwrap();
        }
        wal.shutdown().unwrap();
    }

    // Recover
    let result = RecoveryManager::recover(path).unwrap();

    assert_eq!(result.entries.len(), 100);
    assert!(result.corrupted_at.is_none());

    // Verify entry data
    for (i, (lsn, entry)) in result.entries.iter().enumerate() {
        assert_eq!(lsn.value(), (i + 1) as u64);
        assert_eq!(entry.txn_id, i as u64);
    }
}

#[test]
fn test_recovery_after_crash() {
    let path = "/tmp/test_recovery_crash";

    // Write entries without waiting for all
    {
        let wal = GroupCommitWal::new(path, Default::default()).unwrap();
        for i in 0..1000 {
            let lsn = wal.append(create_test_entry(i)).unwrap();

            // Only wait for first 500
            if i < 500 {
                futures::executor::block_on(wal.wait_for_lsn(lsn)).unwrap();
            }
        }

        // Simulate crash (don't call shutdown)
        std::mem::forget(wal);
    }

    // Recover
    let result = RecoveryManager::recover(path).unwrap();

    // Should recover at least the 500 we waited for
    assert!(result.entries.len() >= 500);
    assert!(result.entries.len() <= 1000);
}

#[test]
fn test_recovery_with_corruption() {
    let path = "/tmp/test_recovery_corrupt";

    // Write entries
    {
        let wal = GroupCommitWal::new(path, Default::default()).unwrap();
        for i in 0..100 {
            let lsn = wal.append(create_test_entry(i)).unwrap();
            futures::executor::block_on(wal.wait_for_lsn(lsn)).unwrap();
        }
        wal.shutdown().unwrap();
    }

    // Corrupt the file
    {
        let mut file = OpenOptions::new()
            .write(true)
            .open(path)
            .unwrap();

        // Seek to middle and write garbage
        file.seek(SeekFrom::Start(5000)).unwrap();
        file.write_all(&[0xFF; 1000]).unwrap();
    }

    // Recover
    let result = RecoveryManager::recover(path).unwrap();

    // Should recover entries before corruption
    assert!(result.entries.len() < 100);
    assert!(result.corrupted_at.is_some());

    // File should be truncated
    let metadata = std::fs::metadata(path).unwrap();
    assert_eq!(metadata.len(), result.last_valid_offset);
}

2.3 Transaction Manager Integration Tests

File: heliosdb-transaction/tests/wal_integration_tests.rs

#[tokio::test]
async fn test_transaction_commit_with_wal() {
    let wal = Arc::new(GroupCommitWal::new(
        "/tmp/test_txn_wal",
        Default::default(),
    ).unwrap());

    let txn_mgr = TransactionCoordinator::new(Arc::clone(&wal));

    // Begin transaction
    let txn_id = txn_mgr.begin().await.unwrap();

    // Perform some writes
    for i in 0..10 {
        txn_mgr.write(txn_id, &format!("key{}", i), &format!("value{}", i))
            .await.unwrap();
    }

    // Commit
    let start = Instant::now();
    txn_mgr.commit(txn_id).await.unwrap();
    let commit_latency = start.elapsed();

    // Should be durable
    let last_lsn = wal.last_flushed_lsn();
    assert!(last_lsn.value() > 0);

    // Commit latency should be reasonable
    assert!(commit_latency < Duration::from_millis(50));
}

#[tokio::test]
async fn test_concurrent_transactions() {
    let wal = Arc::new(GroupCommitWal::new(
        "/tmp/test_concurrent_txns",
        Default::default(),
    ).unwrap());

    let txn_mgr = Arc::new(TransactionCoordinator::new(Arc::clone(&wal)));

    let mut handles = vec![];

    // Spawn 50 concurrent transactions
    for i in 0..50 {
        let mgr = Arc::clone(&txn_mgr);
        handles.push(tokio::spawn(async move {
            let txn_id = mgr.begin().await.unwrap();

            for j in 0..10 {
                mgr.write(txn_id, &format!("key{}_{}", i, j), &format!("value{}", j))
                    .await.unwrap();
            }

            mgr.commit(txn_id).await.unwrap();
        }));
    }

    for h in handles {
        h.await.unwrap();
    }

    // All 50 transactions should be committed
    let metrics = wal.metrics();
    assert!(metrics.total_appends >= 50); // At least 50 commits
}

---

3. Performance Tests

3.1 Throughput Benchmarks

File: benches/group_commit_throughput.rs

use criterion::{black_box, criterion_group, criterion_main, Criterion, Throughput};

fn benchmark_append_throughput(c: &mut Criterion) {
    let mut group = c.benchmark_group("append_throughput");
    group.throughput(Throughput::Elements(1));

    let wal = GroupCommitWal::new("/tmp/bench", Default::default()).unwrap();

    group.bench_function("append_only", |b| {
        b.iter(|| {
            black_box(wal.append(create_test_entry(1)).unwrap());
        });
    });

    group.finish();
}

fn benchmark_commit_throughput(c: &mut Criterion) {
    let mut group = c.benchmark_group("commit_throughput");
    group.throughput(Throughput::Elements(1));

    let runtime = tokio::runtime::Runtime::new().unwrap();
    let wal = GroupCommitWal::new("/tmp/bench", Default::default()).unwrap();

    group.bench_function("append_and_wait", |b| {
        b.to_async(&runtime).iter(|| async {
            let lsn = wal.append(create_test_entry(1)).unwrap();
            black_box(wal.wait_for_lsn(lsn).await.unwrap());
        });
    });

    group.finish();
}

criterion_group!(benches, benchmark_append_throughput, benchmark_commit_throughput);
criterion_main!(benches);

3.2 Latency Benchmarks

File: benches/group_commit_latency.rs

fn benchmark_latency_percentiles(c: &mut Criterion) {
    let mut group = c.benchmark_group("latency");

    let runtime = tokio::runtime::Runtime::new().unwrap();
    let wal = GroupCommitWal::new("/tmp/bench", Default::default()).unwrap();

    group.bench_function("commit_latency", |b| {
        b.to_async(&runtime).iter(|| async {
            let start = Instant::now();
            let lsn = wal.append(create_test_entry(1)).unwrap();
            wal.wait_for_lsn(lsn).await.unwrap();
            black_box(start.elapsed());
        });
    });

    group.finish();
}

// After running benchmarks, analyze percentiles manually
fn analyze_latency_distribution() {
    let runtime = tokio::runtime::Runtime::new().unwrap();
    let wal = GroupCommitWal::new("/tmp/bench", Default::default()).unwrap();

    let mut latencies = vec![];

    for _ in 0..10_000 {
        let latency = runtime.block_on(async {
            let start = Instant::now();
            let lsn = wal.append(create_test_entry(1)).unwrap();
            wal.wait_for_lsn(lsn).await.unwrap();
            start.elapsed()
        });

        latencies.push(latency.as_micros());
    }

    latencies.sort();

    println!("P50: {}μs", latencies[5000]);
    println!("P90: {}μs", latencies[9000]);
    println!("P99: {}μs", latencies[9900]);
    println!("P99.9: {}μs", latencies[9990]);
    println!("Max: {}μs", latencies[9999]);
}

3.3 Regression Tests

File: benches/group_commit_regression.rs

// Compare against baseline (no group commit)

fn benchmark_regression(c: &mut Criterion) {
    let mut group = c.benchmark_group("regression");

    // Baseline: synchronous WAL
    let sync_wal = SynchronousWal::new("/tmp/baseline").unwrap();
    group.bench_function("baseline_sync_wal", |b| {
        b.iter(|| {
            black_box(sync_wal.append(create_test_entry(1)).unwrap());
        });
    });

    // Group commit WAL
    let group_wal = GroupCommitWal::new("/tmp/groupcommit", Default::default()).unwrap();
    group.bench_function("group_commit_wal", |b| {
        b.iter(|| {
            black_box(group_wal.append(create_test_entry(1)).unwrap());
        });
    });

    group.finish();
}

// Expected results:
// - Group commit append should be 10-100x faster
// - Group commit throughput should be 5-10x higher

---

4. Chaos Tests

4.1 Crash Injection Tests

File: tests/chaos/crash_injection.rs

// Note: These tests require special setup (e.g., Docker, VM)

#[test]
#[ignore] // Run separately
fn test_crash_during_write() {
    // This test simulates a crash during write phase
    // Implementation varies by platform
}

#[test]
#[ignore]
fn test_crash_during_fsync() {
    // Simulate crash during fsync
    // Verify recovery handles this correctly
}

#[test]
#[ignore]
fn test_power_failure_simulation() {
    // Simulate sudden power loss
    // Verify no corruption
}

4.2 Failure Injection Tests

File: tests/chaos/failure_injection.rs

struct FaultyFile {
    inner: File,
    fail_rate: f64,
}

impl FaultyFile {
    fn sync_all(&mut self) -> io::Result<()> {
        if rand::random::<f64>() < self.fail_rate {
            Err(io::Error::new(io::ErrorKind::Other, "Simulated fsync failure"))
        } else {
            self.inner.sync_all()
        }
    }
}

#[tokio::test]
async fn test_fsync_failures() {
    // Use FaultyFile to inject fsync failures
    // Verify:
    // - Waiters are notified of failure
    // - Transactions are aborted
    // - System remains consistent
}

#[tokio::test]
async fn test_disk_full() {
    // Simulate disk full condition
    // Verify graceful handling
}

---

5. Test Execution Plan

Phase 1: Development (During implementation)

# Run unit tests continuously
cargo watch -x test

# Run specific test suite
cargo test --package heliosdb-storage --test group_commit_tests

Phase 2: Integration (After implementation)

# Run all integration tests
cargo test --test '*_integration_*'

# Run with logging
RUST_LOG=debug cargo test --test group_commit_integration_tests

# Run with sanitizers
RUSTFLAGS="-Z sanitizer=address" cargo test

Phase 3: Performance (Before release)

# Run benchmarks
cargo bench --bench group_commit_benchmarks

# Compare with baseline
cargo bench --bench group_commit_benchmarks -- --save-baseline main
cargo bench --bench group_commit_benchmarks -- --baseline main

# Flamegraph profiling
cargo flamegraph --bench group_commit_benchmarks

Phase 4: Chaos (Before production)

# Run chaos tests
cargo test --test crash_injection -- --ignored --test-threads=1

# Run with Jepsen (if available)
cd jepsen && lein run test --workload wal-group-commit

---

6. Continuous Integration

CI Pipeline Configuration

.github/workflows/group_commit_tests.yml

name: Group Commit WAL Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run unit tests
        run: cargo test --package heliosdb-storage --lib

  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run integration tests
        run: cargo test --test '*_integration_*'

  performance-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run benchmarks
        run: cargo bench --bench group_commit_benchmarks -- --save-baseline ci

  code-coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install tarpaulin
        run: cargo install cargo-tarpaulin
      - name: Generate coverage
        run: cargo tarpaulin --out Xml
      - name: Upload to codecov
        uses: codecov/codecov-action@v2

---

7. Acceptance Criteria

Functional Correctness

• All unit tests pass (>80% coverage)
• All integration tests pass
• Recovery handles all failure modes correctly
• Durability guarantees maintained

Performance Requirements

• Throughput ≥ 10,000 commits/sec (HDD)
• P99 latency ≤ 20ms
• Fsync reduction ≥ 90%
• No regression in read path

Reliability

• Crash recovery works correctly
• No data corruption under concurrent load
• Graceful degradation under failures
• No memory leaks (valgrind clean)

Production Readiness

• Chaos tests pass
• Performance benchmarks meet targets
• CI pipeline green
• Code coverage ≥ 80%

---

8. Test Metrics Dashboard

Track these metrics throughout development:

Metric	Target	Current	Status
Unit test coverage	≥80%	TBD	🟡
Integration tests	All passing	TBD	🟡
Throughput (HDD)	≥10K/sec	TBD	🟡
P99 latency	≤20ms	TBD	🟡
Fsync reduction	≥90%	TBD	🟡
Memory leaks	0	TBD	🟡

Legend: 🟢 Passing | 🟡 In Progress | 🔴 Failing

---

Next Steps

1. Set up test infrastructure
2. Implement unit tests (Day 1-2)
3. Implement integration tests (Day 3)
4. Run performance benchmarks (Day 5)
5. Set up CI pipeline
6. Monitor test metrics continuously

---

Document Status: Test Plan Ready Owner: QA Team + Development Team Review Date: 2025-11-10