---
title: "Flink CDC Integration for HeliosDB"
---

# Flink CDC Integration for HeliosDB

:::caution[Planned: Flink CDC integration]
The Flink CDC integration for HeliosDB Full is planned. The current release includes no
Flink connector or state backend; this page describes the planned design. To see recent
changes today, run `SHOW CDC EVENTS [LIMIT n]` (see the
[CDC tutorial](/docs/full/tutorials/intermediate/05-cdc-events/)).
:::

Planned design for using Apache Flink CDC (Change Data Capture) with HeliosDB for real-time data streaming.

## Overview

The planned HeliosDB Flink CDC integration provides:

- **Debezium-compatible CDC** - Capture changes from databases in real-time
- **Exactly-once semantics** - Guaranteed delivery without duplicates
- **Low latency** - <100ms CDC latency for change capture
- **High throughput** - 50K+ ops/sec state backend performance
- **HeliosDB State Backend** - RocksDB alternative with better performance

## Features

### 1. CDC Source Connector

The CDC source connector captures database changes using logical replication.

### 2. Snapshot Modes

Control initial snapshot behavior:

#### Always
Always take full snapshot on startup.

#### Initial
Take snapshot only if no previous checkpoint exists.

#### Never
Skip snapshot, only capture incremental changes.

#### WhenNeeded
Take snapshot only when schema changes detected.

### 3. Exactly-Once Semantics

Ensure exactly-once delivery with checkpointing.

### 4. Checkpoint Management

Manage CDC checkpoints for recovery.

### 5. Debezium Format

Use Debezium-compatible format for interoperability.

### 6. Monitoring and Metrics

Track CDC performance: events processed (creates, updates, deletes), snapshot records, and lag (`lag_ms`).

## Performance

### CDC Latency

- **Target**: <100ms end-to-end latency
- **Measured**: ~50ms p99 latency for typical workloads
- **Factors**: Network latency, poll interval, processing time

### Throughput

- **Events/sec**: 10K-50K events per second
- **Batching**: Configurable buffer size for optimal throughput
- **Backpressure**: Automatic backpressure handling

## Configuration Guide

### PostgreSQL Setup

1. **Enable logical replication**:
```sql
-- postgresql.conf
wal_level = logical
max_replication_slots = 4
max_wal_senders = 4
```

2. **Create publication**:
```sql
CREATE PUBLICATION helios_pub FOR TABLE users, orders;
```

3. **Create replication slot**:
```sql
SELECT pg_create_logical_replication_slot('helios_cdc_slot', 'pgoutput');
```

4. **Grant permissions**:
```sql
GRANT SELECT ON users, orders TO cdc_user;
GRANT REPLICATION ON DATABASE mydb TO cdc_user;
```

### Connection String Format

```
postgres://user:password@host:port/database?replication=database
```

### Tuning Parameters

#### Buffer Size
- Larger = better throughput, more memory
- Smaller = lower latency, less memory

#### Poll Interval
- Shorter = lower latency, more CPU
- Longer = less CPU, higher latency

#### Heartbeat Interval
- Prevents idle connection timeout
- Updates checkpoint even when no changes

## Best Practices

### 1. Checkpoint Frequency

Balance durability vs performance: checkpointing every second adds high overhead and every 10 minutes leads to long recovery; an interval of around 60 seconds is recommended.

### 2. Table Selection

Monitor specific tables to reduce overhead; don't monitor audit/log tables.

### 3. Error Handling

Implement retry logic with exponential backoff when event processing fails.

### 4. Schema Evolution

Handle schema changes gracefully: when a schema-change event arrives, update the downstream schema.

## Troubleshooting

### High Lag

**Symptoms**: `lag_ms` metric increasing
**Solutions**:
- Increase buffer size
- Optimize event processing
- Scale horizontally with partitioning

### Missed Events

**Symptoms**: Gap in LSN sequence
**Solutions**:
- Check replication slot active
- Verify publication includes all tables
- Review database logs for errors

### Memory Growth

**Symptoms**: Increasing memory usage
**Solutions**:
- Reduce buffer size
- Implement backpressure
- Add event filtering

## Comparison: HeliosDB vs Debezium

| Feature | HeliosDB CDC | Debezium |
|---------|-------------|----------|
| Latency | <100ms | ~200ms |
| Throughput | 50K ops/sec | 30K ops/sec |
| Memory | Lower | Higher |
| Setup | Simpler | More complex |
| Ecosystem | Rust-native | Java/Kafka |

## Next Steps

- See State Backend Guide for state management
- See Table API Guide for SQL integration