Multi-Region Operations Runbook

HeliosDB Multi-Region Operations Runbook

Use this runbook for regional health, failover, consistency checks, and cross-region replication.

Region Health Monitoring

Monitor each region for:

• Health endpoint status.
• Query latency.
• Write availability in the primary region.
• Replica replay lag in secondary regions.
• Error rate and network loss.

for region in us-east-1 us-west-2 eu-west-1; do
  curl -f "http://heliosdb-${region}:7000/health"
done

Manual Failover Procedure

Use manual failover only when the primary region cannot serve safely or planned maintenance requires promotion.

1. Confirm primary region impact.
2. Freeze or drain writes if possible.
3. Select the healthiest secondary region.
4. Confirm replication lag and data consistency.
5. Promote the selected region.
6. Update routing.
7. Validate writes and reads.
8. Monitor for split-brain risk.

Consistency Verification

Before and after failover:

SELECT count(*) FROM critical_table;
SELECT max(updated_at) FROM critical_table;

Compare checksums or application-level invariants for critical datasets.

Cross-Region Replication Checks

Check lag and replay state:

SELECT application_name, state, sent_lsn, replay_lsn
FROM pg_stat_replication;

Alert when lag is sustained beyond the recovery point objective.

Region Addition/Removal

To add a region:

1. Provision infrastructure and network access.
2. Seed data from a verified backup.
3. Start replication.
4. Validate read-only traffic.
5. Add the region to routing gradually.

To remove a region:

1. Drain traffic.
2. Confirm no primary responsibilities remain.
3. Stop replication.
4. Archive logs and metrics.
5. Decommission resources.

Multi-Region Troubleshooting

Symptom	Likely Cause	Action
High replication lag	Network or write volume spike	Reduce noncritical writes and inspect network
Region health flapping	Load balancer or network instability	Remove region from routing until stable
Read inconsistency	Stale replica	Route critical reads to primary
Split-brain risk	Conflicting promotion	Stop writes and reconcile leadership