---
title: "Incident Response Runbook"
---

# HeliosDB Incident Response Runbook

Use this runbook for service outages, data integrity alerts, severe latency, failed backups, and security-sensitive events.

## Incident Classification

| Severity | Criteria | Initial Response |
|---|---|---|
| P0 | Complete outage, data loss, corruption, or confirmed security incident | Immediate response and executive escalation |
| P1 | Major degradation, regional outage, or core feature unavailable | Incident lead and senior SRE engaged |
| P2 | Partial degradation with workaround | On-call owner investigates and communicates |
| P3 | Minor issue with limited impact | Track and resolve in normal priority |
| P4 | Informational alert or maintenance notice | Record and monitor |

## Initial Response Steps

1. Acknowledge the alert.
2. Assign an incident lead.
3. Record the start time, symptoms, and known customer impact.
4. Open the incident communication channel.
5. Check service health, logs, and recent changes.
6. Stabilize before making broad changes.

Useful checks:

```bash
curl -f http://heliosdb-lb:7000/health
psql -h heliosdb-lb -U admin -c "SELECT now(), version();"
psql -h heliosdb-lb -U admin -c "SELECT pid, state, wait_event_type, query FROM pg_stat_activity LIMIT 20;"
```

## Escalation Procedures

Escalate when:

- Impact is P0 or P1.
- The service is not stabilizing within the expected response window.
- Data loss, corruption, or unauthorized access is suspected.
- The runbook procedure fails or produces unexpected results.
- Multiple teams or infrastructure providers are involved.

Escalation handoff must include:

- Incident severity and impact.
- Timeline of actions already taken.
- Current dashboards, logs, and hypotheses.
- Rollback or mitigation options.

## Communication Templates

Initial update:

```text
We are investigating elevated errors or latency for HeliosDB. The incident team is active and will provide updates as more information is available.
```

Mitigation update:

```text
The team has identified the likely cause and is applying mitigation. We are monitoring recovery and will provide the next update after validation.
```

Resolution update:

```text
The issue is resolved and HeliosDB health has returned to normal. A follow-up review will capture root cause and prevention items.
```

## Postmortem Process

For P0 and P1 incidents:

1. Preserve logs, metrics, and timeline.
2. Document impact, trigger, detection, mitigation, and resolution.
3. Identify corrective actions with owners.
4. Update runbooks, alerts, and tests.
5. Review completion of follow-up actions.

## Incident Examples

### Complete Service Outage

1. Confirm load balancer and node health.
2. Check recent deployments and configuration changes.
3. Restore traffic to a known healthy node or region.
4. Roll back if the outage follows a deployment.

### Replication Lag

1. Check primary write rate and replica replay lag.
2. Confirm network health between regions.
3. Reduce nonessential write load if needed.
4. Rebuild or reseed replica only after preserving diagnostics.

### Disk Space Exhaustion

1. Stop nonessential writes if the primary is at risk.
2. Check WAL, logs, temporary files, and backup staging directories.
3. Add capacity or move cold data.
4. Validate checkpoints and resume normal traffic.