---
title: "CDC Webhooks Troubleshooting Guide"
---

# CDC Webhooks Troubleshooting Guide

:::caution[Planned: CDC webhooks]
CDC webhooks are planned for HeliosDB Full. The current release does not capture
changes from the write-ahead log for delivery or call HTTP endpoints, and it does not
accept the CDC stream statements on this page yet. This page describes the planned design. Today, `SHOW CDC EVENTS [LIMIT n]` returns the most recent
INSERT, UPDATE and DELETE events from the server's in-memory change-event log (see the
[CDC tutorial](/docs/full/tutorials/intermediate/05-cdc-events/)).
:::

---

This guide covers common issues encountered when working with CDC webhooks and their solutions.

## Table of Contents

1. [Events Not Being Captured](#events-not-being-captured)
2. [Webhook Delivery Failures](#webhook-delivery-failures)
3. [High Latency](#high-latency)
4. [Circuit Breaker Issues](#circuit-breaker-issues)
5. [Duplicate Events](#duplicate-events)
6. [Queue Backlog](#queue-backlog)
7. [Security Issues](#security-issues)
8. [Monitoring and Diagnostics](#monitoring-and-diagnostics)

---

## Events Not Being Captured

### Symptom

Database changes are not triggering webhook events.

### Diagnosis

```sql
-- Check if CDC stream exists and is active
SHOW CDC STREAMS;

-- Check stream status
SHOW CDC STREAM STATUS your_stream_name;
```

### Solutions

**1. Stream is paused**

```sql
-- Resume the stream
ALTER CDC STREAM your_stream_name RESUME;
```

**2. Table not included in stream**

```sql
-- Check stream configuration
SHOW CDC STREAM STATUS your_stream_name;

-- Create new stream with correct table
CREATE CHANGE DATA CAPTURE ON your_table
  TO KAFKA 'localhost:9092' TOPIC 'your-topic'
  FORMAT JSON;
```

**3. Filter excluding events**

```yaml
# Check filter configuration
filters:
  - name: your_filter
    tables:
      - your_table  # Ensure table is included
    operations:
      - INSERT
      - UPDATE
      - DELETE  # Ensure operation is included
```

---

## Webhook Delivery Failures

### Symptom

Events are captured but not delivered to webhook endpoints.

### Diagnosis

```sql
-- Check stream status for errors
SHOW CDC STREAM STATUS your_stream_name;
```

```promql
# Check Prometheus metrics
rate(heliosdb_webhook_delivery_total{status="failure"}[5m])
```

### Solutions

**1. Endpoint unreachable**

```bash
# Test endpoint connectivity
curl -X POST https://your-endpoint.com/webhooks \
  -H "Content-Type: application/json" \
  -d '{"test": true}'
```

```yaml
# Verify endpoint URL in configuration
webhooks:
  - url: https://your-endpoint.com/webhooks  # Check URL
```

**2. SSL/TLS certificate issues**

For production, ensure certificates are valid:

```bash
# Check certificate
openssl s_client -connect your-endpoint.com:443
```

**3. Endpoint returning errors**

Check your endpoint logs for error details.

```python
# Ensure endpoint returns 2xx status
def webhook_handler(request):
    try:
        process_event(request.json)
        return {"status": "success"}, 200  # Return 2xx
    except Exception as e:
        return {"error": str(e)}, 500  # Will trigger retry
```

**4. Timeout issues**

```yaml
# Increase timeout if endpoint is slow
delivery:
  timeout_secs: 60  # Increase from default 30
```

**5. Retry exhausted**

```yaml
# Increase retries
delivery:
  max_retries: 5  # Increase from default 3
```

---

## High Latency

### Symptom

Events are being delivered but with high latency (>100ms P99).

### Diagnosis

```promql
# Check latency metrics
histogram_quantile(0.99, rate(heliosdb_webhook_delivery_duration_seconds_bucket[5m]))
```

### Solutions

**1. Endpoint is slow**

```bash
# Measure endpoint response time
time curl -X POST https://your-endpoint.com/webhooks \
  -H "Content-Type: application/json" \
  -d '{"test": true}'
```

Consider:
- Optimizing endpoint processing
- Async processing (acknowledge quickly, process later)
- Adding more endpoint instances

**2. Network latency**

```bash
# Check network latency
ping your-endpoint.com
traceroute your-endpoint.com
```

Consider:
- Deploying HeliosDB closer to endpoints
- Using a faster network path
- Adding regional endpoint replicas

**3. Worker pool too small**

```yaml
# Increase worker pool
worker_pool:
  num_workers: 200  # Increase from default
  max_concurrent_requests: 2000
```

**4. Queue backlog**

```promql
# Check queue size
heliosdb_event_queue_size
```

See [Queue Backlog](#queue-backlog) section.

---

## Circuit Breaker Issues

### Symptom

Circuit breaker is open, blocking delivery to healthy endpoints.

### Diagnosis

```promql
# Check circuit state
heliosdb_circuit_breaker_state == 2  # 2 = Open
```

### Solutions

**1. Circuit opened due to transient failures**

Wait for timeout period, circuit will transition to half-open:

```yaml
circuit_breaker:
  timeout_secs: 60  # Circuit will try again after 60s
```

**2. False positives from timeout**

```yaml
# Increase timeout before marking as failure
delivery:
  timeout_secs: 60  # Give endpoint more time
```

**3. Adjust circuit breaker sensitivity**

```yaml
circuit_breaker:
  failure_threshold: 10  # Increase from 5
  success_threshold: 3   # Require more successes to close
```

---

## Duplicate Events

### Symptom

Same events are being delivered multiple times.

### Diagnosis

Check if your endpoint is receiving duplicate event IDs.

### Solutions

**1. Verify exactly-once is enabled**

```yaml
exactly_once:
  delivery_window_secs: 86400  # 24 hour window
```

**2. Implement idempotency in receiver**

```python
seen_events = set()

def handle_webhook(event):
    event_id = event['event_id']

    if event_id in seen_events:
        return {"status": "duplicate"}, 200

    seen_events.add(event_id)
    process_event(event)

    return {"status": "success"}, 200
```

**3. Use database for deduplication**

```sql
-- Create dedup table
CREATE TABLE webhook_events (
    event_id VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP DEFAULT NOW()
);

-- Check before processing
INSERT INTO webhook_events (event_id)
VALUES ('evt_123')
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
```

**4. Check for retry storm**

```promql
# High retry rate indicates issues
rate(heliosdb_webhook_retry_total[5m]) > 100
```

---

## Queue Backlog

### Symptom

Events are queuing up faster than they can be delivered.

### Diagnosis

```promql
# Check queue size
heliosdb_event_queue_size{queue_type="fast"}
heliosdb_event_queue_size{queue_type="overflow"}
```

### Solutions

**1. Increase worker pool**

```yaml
worker_pool:
  num_workers: 200
  max_concurrent_requests: 5000
  connection_pool_size: 1000
```

**2. Add more endpoints (load balance)**

```yaml
webhooks:
  - url: https://endpoint-1.example.com/webhooks
  - url: https://endpoint-2.example.com/webhooks
  - url: https://endpoint-3.example.com/webhooks
```

**3. Reduce event volume with filters**

```yaml
filters:
  - name: reduce_volume
    tables:
      - important_table  # Only essential tables
    operations:
      - INSERT  # Skip updates if not needed
```

**4. Batch events (if supported by receiver)**

```yaml
delivery:
  batch_size: 100  # Send 100 events per request
  batch_timeout_ms: 1000  # Or after 1 second
```

**5. Scale horizontally**

Deploy multiple CDC processors with table sharding.

---

## Security Issues

### Symptom

Webhook signature verification failing or security alerts.

### Diagnosis

```python
# Check signature header
signature = request.headers.get('X-HeliosDB-Signature')
print(f"Received signature: {signature}")
```

### Solutions

**1. Signature verification failing**

```python
import hmac
import hashlib

def verify_signature(payload, signature, secret):
    # Ensure payload is bytes
    if isinstance(payload, str):
        payload = payload.encode('utf-8')

    expected = 'sha256=' + hmac.new(
        secret.encode('utf-8'),
        payload,
        hashlib.sha256
    ).hexdigest()

    # Use constant-time comparison
    return hmac.compare_digest(signature, expected)
```

**2. Wrong secret key**

```yaml
# Verify secret matches between sender and receiver
security:
  hmac_secret_env: WEBHOOK_SECRET  # Check this environment variable
```

**3. Payload modification (proxy/load balancer)**

Ensure no proxy or load balancer is modifying the request body.

**4. TLS certificate issues**

```bash
# Verify certificate
openssl verify -CAfile ca.pem cert.pem

# Check certificate chain
openssl s_client -connect your-endpoint.com:443 -showcerts
```

---

## Monitoring and Diagnostics

### Check Overall Health

```sql
-- Stream overview
SHOW CDC STREAMS;
```

```promql
# Overall delivery success rate
sum(rate(heliosdb_webhook_delivery_total{status="success"}[5m])) /
sum(rate(heliosdb_webhook_delivery_total[5m])) * 100
```

### Check Specific Endpoint

```promql
# Delivery rate for specific endpoint
rate(heliosdb_webhook_delivery_total{webhook_url="https://example.com"}[5m])

# Error rate for specific endpoint
rate(heliosdb_webhook_delivery_total{webhook_url="https://example.com",status="failure"}[5m])
```

### Debug Logging

```yaml
# Enable debug logging
logging:
  level: debug
  modules:
    heliosdb_cdc: debug
    heliosdb_webhooks: debug
```

---

## Common Error Messages

| Error | Cause | Solution |
|-------|-------|----------|
| `Connection refused` | Endpoint not listening | Check endpoint is running |
| `Connection timed out` | Network/firewall issue | Check network connectivity |
| `SSL certificate error` | Invalid/expired cert | Update certificate |
| `429 Too Many Requests` | Rate limited | Reduce delivery rate |
| `503 Service Unavailable` | Endpoint overloaded | Scale endpoint |
| `Circuit breaker open` | Too many failures | Wait for recovery |
| `Queue overflow` | Backlog too large | Scale workers |

---

## Related Documentation

- [README.md](https://www.heliosdb.com/docs/full/features/webhooks/readme/) - Feature overview
- [USER_GUIDE.md](https://www.heliosdb.com/docs/full/features/webhooks/user_guide/) - Comprehensive guide