Incident Response Runbook

HeliosDB Incident Response Runbook

Use this runbook for service outages, data integrity alerts, severe latency, failed backups, and security-sensitive events.

Incident Classification

Severity	Criteria	Initial Response
P0	Complete outage, data loss, corruption, or confirmed security incident	Immediate response and executive escalation
P1	Major degradation, regional outage, or core feature unavailable	Incident lead and senior SRE engaged
P2	Partial degradation with workaround	On-call owner investigates and communicates
P3	Minor issue with limited impact	Track and resolve in normal priority
P4	Informational alert or maintenance notice	Record and monitor

Initial Response Steps

1. Acknowledge the alert.
2. Assign an incident lead.
3. Record the start time, symptoms, and known customer impact.
4. Open the incident communication channel.
5. Check service health, logs, and recent changes.
6. Stabilize before making broad changes.

Useful checks:

curl -f http://heliosdb-lb:7000/health
psql -h heliosdb-lb -U admin -c "SELECT now(), version();"
psql -h heliosdb-lb -U admin -c "SELECT pid, state, wait_event_type, query FROM pg_stat_activity LIMIT 20;"

Escalation Procedures

Escalate when:

• Impact is P0 or P1.
• The service is not stabilizing within the expected response window.
• Data loss, corruption, or unauthorized access is suspected.
• The runbook procedure fails or produces unexpected results.
• Multiple teams or infrastructure providers are involved.

Escalation handoff must include:

• Incident severity and impact.
• Timeline of actions already taken.
• Current dashboards, logs, and hypotheses.
• Rollback or mitigation options.

Communication Templates

Initial update:

We are investigating elevated errors or latency for HeliosDB. The incident team is active and will provide updates as more information is available.

Mitigation update:

The team has identified the likely cause and is applying mitigation. We are monitoring recovery and will provide the next update after validation.

Resolution update:

The issue is resolved and HeliosDB health has returned to normal. A follow-up review will capture root cause and prevention items.

Postmortem Process

For P0 and P1 incidents:

1. Preserve logs, metrics, and timeline.
2. Document impact, trigger, detection, mitigation, and resolution.
3. Identify corrective actions with owners.
4. Update runbooks, alerts, and tests.
5. Review completion of follow-up actions.

Incident Examples

Complete Service Outage

1. Confirm load balancer and node health.
2. Check recent deployments and configuration changes.
3. Restore traffic to a known healthy node or region.
4. Roll back if the outage follows a deployment.

Replication Lag

1. Check primary write rate and replica replay lag.
2. Confirm network health between regions.
3. Reduce nonessential write load if needed.
4. Rebuild or reseed replica only after preserving diagnostics.

Disk Space Exhaustion

1. Stop nonessential writes if the primary is at risk.
2. Check WAL, logs, temporary files, and backup staging directories.
3. Add capacity or move cold data.
4. Validate checkpoints and resume normal traffic.