---
title: "Multi-Region Operations Runbook"
---

# HeliosDB Multi-Region Operations Runbook

Use this runbook for regional health, failover, consistency checks, and cross-region replication.

## Region Health Monitoring

Monitor each region for:

- Health endpoint status.
- Query latency.
- Write availability in the primary region.
- Replica replay lag in secondary regions.
- Error rate and network loss.

```bash
for region in us-east-1 us-west-2 eu-west-1; do
  curl -f "http://heliosdb-${region}:7000/health"
done
```

## Manual Failover Procedure

Use manual failover only when the primary region cannot serve safely or planned maintenance requires promotion.

1. Confirm primary region impact.
2. Freeze or drain writes if possible.
3. Select the healthiest secondary region.
4. Confirm replication lag and data consistency.
5. Promote the selected region.
6. Update routing.
7. Validate writes and reads.
8. Monitor for split-brain risk.

## Consistency Verification

Before and after failover:

```sql
SELECT count(*) FROM critical_table;
SELECT max(updated_at) FROM critical_table;
```

Compare checksums or application-level invariants for critical datasets.

## Cross-Region Replication Checks

Check lag and replay state:

```sql
SELECT application_name, state, sent_lsn, replay_lsn
FROM pg_stat_replication;
```

Alert when lag is sustained beyond the recovery point objective.

## Region Addition/Removal

To add a region:

1. Provision infrastructure and network access.
2. Seed data from a verified backup.
3. Start replication.
4. Validate read-only traffic.
5. Add the region to routing gradually.

To remove a region:

1. Drain traffic.
2. Confirm no primary responsibilities remain.
3. Stop replication.
4. Archive logs and metrics.
5. Decommission resources.

## Multi-Region Troubleshooting

| Symptom | Likely Cause | Action |
|---|---|---|
| High replication lag | Network or write volume spike | Reduce noncritical writes and inspect network |
| Region health flapping | Load balancer or network instability | Remove region from routing until stable |
| Read inconsistency | Stale replica | Route critical reads to primary |
| Split-brain risk | Conflicting promotion | Stop writes and reconcile leadership |