Deployment Runbook

HeliosDB Deployment Runbook

Use this runbook for planned HeliosDB version updates, configuration changes, and rollback decisions.

Pre-Deployment Checklist

Before changing production, confirm:

• The target build or image tag is approved.
• Staging deployment and smoke tests passed.
• Backups are recent and restore verification has succeeded.
• The rollback version, configuration, and owner are identified.
• Required migrations are backward compatible or have a tested maintenance window.
• Monitoring dashboards and alert routing are available.

Rolling Update Procedure

Use rolling updates when the change is backward compatible and the cluster can run mixed versions temporarily.

1. Disable unrelated automation that could change capacity during the rollout.
2. Remove one node from serving traffic.
3. Wait for active sessions to drain.
4. Apply the binary, image, or configuration update.
5. Restart the node.
6. Run health checks and a representative query set.
7. Return the node to service.
8. Repeat one node at a time.

Validation commands:

curl -f http://heliosdb-node:7000/health
psql -h heliosdb-lb -U admin -c "SELECT version();"
psql -h heliosdb-lb -U admin -c "SELECT count(*) FROM pg_stat_activity;"

Blue-Green Deployment

Use blue-green deployment for major changes, schema changes that cannot run mixed, or risky configuration updates.

1. Provision the green environment with the target release.
2. Restore or replicate production data into green.
3. Run migration checks, read/write smoke tests, and performance probes.
4. Shift a small percentage of read traffic to green.
5. Move all traffic after validation.
6. Keep blue available until the rollback window closes.

Rollback Procedure

Rollback immediately when health checks fail, error rates rise above the rollback threshold, or customer-facing latency exceeds the agreed limit.

1. Stop the rollout.
2. Remove updated nodes from traffic.
3. Reapply the previous binary, image, and configuration.
4. Restart and validate each node.
5. Confirm replication and query health.
6. Record the failure mode for follow-up.

Post-Deployment Validation

Run these checks after rollout:

• Cluster health endpoint returns success.
• Write, read, and transaction smoke tests pass.
• Replication lag is within the normal range.
• Error rate and latency remain within baseline.
• Logs do not show repeated startup, migration, or connection errors.

Common Deployment Issues

Symptom	Likely Cause	Action
Health check fails after restart	Configuration error	Revert config and restart
Node rejoins but receives no traffic	Load balancer or service selector mismatch	Validate service routing
Replication lag grows	Updated node cannot keep up	Pause rollout and investigate I/O
Migration fails	Incompatible schema change	Roll back and rerun migration plan in staging