---
title: "HeliosDB-Lite High Availability Hands-On Tutorial"
description: "Build a HeliosDB Lite primary with a sync and an async standby behind HeliosProxy, test write routing and read routing, monitor replication in SQL."
---

# HeliosDB-Lite High Availability Hands-On Tutorial

This tutorial builds a three-node HeliosDB Lite cluster (one primary, one synchronous
standby, one asynchronous standby) with HeliosProxy in front of it, then walks through
write routing, read routing, replication monitoring and what happens when nodes stop.
Each step uses the `heliosdb-lite start` flags, SQL statements and HeliosProxy settings
described here; steps for capabilities that are planned are marked as planned.

## Before You Start

Replication is an opt-in build feature of the licensed source distribution. A binary
built with the `ha-tier1` feature serves the replication flags; the default build and
the stock Dockerfile of the distribution do not include it, and the source distribution
marks the feature as experimental. A binary built without it refuses to start with any
`--replication-role` other than `standalone`.

| Capability | Status |
|------------|--------|
| WAL streaming from a primary to standbys | Available (`ha-tier1` build) |
| `--sync-mode` `async`, `semi-sync`, `sync` | Available; see [Sync Modes](#sync-modes) |
| Write forwarding from `sync` and `semi-sync` standbys | Available, with the limits in [Part 3](#part-3-write-routing) |
| Replication views (`pg_replication_*`), `SHOW TOPOLOGY`, `SHOW CLUSTER STATUS`, `SWITCHOVER CHECK` | Available (`ha-tier1` build) |
| Controlled switchover (`SWITCHOVER TO`) | Planned: the statement is accepted but does not change roles |
| Automatic failover and promotion | Planned: promotion is a manual or external step |
| Observer nodes for split-brain protection | Planned: the `observer` role starts no service |
| Routing, health checks and write buffering in front of the cluster | HeliosProxy, a separate Apache-2.0 product |

The examples use the server's default `trust` authentication, because write forwarding
from a standby connects to the primary without a password (see [Part 3](#part-3-write-routing)).
Keep every port on loopback or on a private network, as the examples do. For password or
SCRAM authentication, send writes to the primary or through HeliosProxy instead of
relying on standby forwarding.

---

## Architecture

```
                    ┌─────────────────────────────────────┐
                    │            APPLICATION              │
                    └─────────────────┬───────────────────┘
                                      │ PostgreSQL protocol
                    ┌─────────────────▼───────────────────┐
                    │            HELIOSPROXY              │
                    │  client port 6432, admin API 9090   │
                    │  writes -> primary                  │
                    │  reads  -> healthy nodes            │
                    │  write buffering: write_timeout_secs│
                    └───────┬───────────┬───────────┬─────┘
                            │           │           │
              ┌─────────────▼───┐   ┌───▼───────┐   ┌───▼─────────────┐
              │     PRIMARY     │   │  STANDBY  │   │    STANDBY      │
              │  (read/write)   │   │   sync    │   │     async       │
              └────────┬────────┘   └─────▲─────┘   └────────▲────────┘
                       │  WAL stream      │                  │
                       └──────────────────┴──────────────────┘
```

Each Lite node serves PostgreSQL on its `--port`, the REST API and `/health` on its
`--http-port`, and the primary accepts standby connections on its `--replication-port`
(default `5433`). Standbys connect to the primary given in `--primary-host`; the primary
needs no list of its standbys.

### Sync Modes

| Mode | On the primary | On a standby |
|------|----------------|--------------|
| `sync` / `semi-sync` | Each write waits until one standby confirms it has received the change. If no standby confirms within 10 seconds, the write completes on the primary and a warning is logged. | Forwards writes to the primary |
| `async` | Writes do not wait for standbys | Rejects writes (SQLSTATE `25006`) |

---

## Part 1: Docker Deployment

### Prerequisites

Docker with the Compose plugin, `psql`, `curl` and `openssl` on the host.

### Step 1: Build an Image with Replication

In the unpacked source distribution, copy the stock Dockerfile with `ha-tier1` added to
its build features, then build the image:

```bash
sed 's/--features "encryption,vector-search"/--features "encryption,vector-search,ha-tier1"/' \
  deployment/docker/Dockerfile > Dockerfile.ha
grep -n 'ha-tier1' Dockerfile.ha    # the cargo build line must now list ha-tier1
docker build -f Dockerfile.ha -t heliosdb-lite:ha .
```

The image runs `heliosdb-lite` as its entrypoint, so the Compose `command` below starts
with the `start` subcommand.

### Step 2: Create the Compose File

Create `docker-compose.ha-cluster.yml`. Host ports are published on `127.0.0.1` only.

```yaml
services:
  primary:
    image: heliosdb-lite:ha
    hostname: primary
    command: ["start", "--data-dir", "/data", "--listen", "0.0.0.0", "--port", "5432",
              "--http-port", "8080",
              "--replication-role", "primary", "--replication-port", "5433",
              "--sync-mode", "sync"]
    environment:
      - HELIOSDB_JWT_SECRET=${HELIOSDB_JWT_SECRET:?set HELIOSDB_JWT_SECRET}
    ports:
      - "127.0.0.1:15432:5432"   # PostgreSQL
      - "127.0.0.1:18080:8080"   # REST API and /health
    volumes:
      - primary-data:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 5s
      timeout: 3s
      retries: 3

  standby-sync:
    image: heliosdb-lite:ha
    hostname: standby-sync
    command: ["start", "--data-dir", "/data", "--listen", "0.0.0.0", "--port", "5432",
              "--http-port", "8080",
              "--replication-role", "standby", "--primary-host", "primary:5433",
              "--sync-mode", "sync"]
    environment:
      - HELIOSDB_JWT_SECRET=${HELIOSDB_JWT_SECRET:?set HELIOSDB_JWT_SECRET}
    ports:
      - "127.0.0.1:15442:5432"
      - "127.0.0.1:18081:8080"
    volumes:
      - standby-sync-data:/data
    depends_on:
      primary:
        condition: service_healthy

  standby-async:
    image: heliosdb-lite:ha
    hostname: standby-async
    command: ["start", "--data-dir", "/data", "--listen", "0.0.0.0", "--port", "5432",
              "--http-port", "8080",
              "--replication-role", "standby", "--primary-host", "primary:5433",
              "--sync-mode", "async"]
    environment:
      - HELIOSDB_JWT_SECRET=${HELIOSDB_JWT_SECRET:?set HELIOSDB_JWT_SECRET}
    ports:
      - "127.0.0.1:15452:5432"
      - "127.0.0.1:18082:8080"
    volumes:
      - standby-async-data:/data
    depends_on:
      primary:
        condition: service_healthy

  proxy:
    # HeliosProxy image; pin a release tag in production
    image: ghcr.io/heliosdatabase/hdb-heliosdb-proxy:latest
    ports:
      - "127.0.0.1:16432:6432"   # PostgreSQL through HeliosProxy
      - "127.0.0.1:19090:9090"   # HeliosProxy admin API
    volumes:
      - ./proxy.toml:/etc/helios/proxy.toml:ro
    depends_on:
      - primary
      - standby-sync
      - standby-async

volumes:
  primary-data:
  standby-sync-data:
  standby-async-data:
```

A standby forwards writes to the PostgreSQL port of the host named in `--primary-host`,
which is taken as `5432` unless the standby's environment sets `HELIOSDB_PRIMARY_PG_PORT`.

### Step 3: Create the HeliosProxy Configuration

Create `proxy.toml` next to the Compose file. The proxy image reads
`/etc/helios/proxy.toml`.

```toml
listen_address = "0.0.0.0:6432"
admin_address = "0.0.0.0:9090"
# An admin API on a non-loopback address needs a token. Generate one with:
#   openssl rand -hex 32
admin_token = "replace-with-a-random-token"
tr_enabled = true
tr_mode = "session"
write_timeout_secs = 30

[pool]
min_connections = 2
max_connections = 100
idle_timeout_secs = 300
max_lifetime_secs = 1800
acquire_timeout_secs = 30
test_on_acquire = true

[load_balancer]
read_strategy = "round_robin"
read_write_split = true
latency_threshold_ms = 100

[health]
check_interval_secs = 5
check_timeout_secs = 3
failure_threshold = 3
success_threshold = 2
check_query = "SELECT 1"

[[nodes]]
name = "primary"
host = "primary"
port = 5432
role = "primary"
weight = 100
enabled = true

[[nodes]]
name = "standby-sync"
host = "standby-sync"
port = 5432
role = "standby"
weight = 100
enabled = true

[[nodes]]
name = "standby-async"
host = "standby-async"
port = 5432
role = "standby"
weight = 100
enabled = true
```

Keep the default session pooling (no `[pool_mode]` section) in front of Lite; see the
[HeliosProxy Integration Guide](/docs/lite/guides/heliosproxy_integration_guide/).

### Step 4: Start the Cluster

```bash
# HELIOSDB_JWT_SECRET is mandatory: compose passes it to every Lite node
export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"

docker compose -f docker-compose.ha-cluster.yml up -d
docker compose -f docker-compose.ha-cluster.yml ps
```

All four services should be running. The standbys start once the primary answers its
`/health` check.

### Step 5: Verify Connectivity and Roles

```bash
# Through HeliosProxy
psql -h 127.0.0.1 -p 16432 -U postgres -c "SELECT 1"

# Each node directly: role and sync mode
psql -h 127.0.0.1 -p 15432 -U postgres -c "SELECT role, sync_mode FROM pg_replication_status"
psql -h 127.0.0.1 -p 15442 -U postgres -c "SELECT role, sync_mode FROM pg_replication_status"
psql -h 127.0.0.1 -p 15452 -U postgres -c "SELECT role, sync_mode FROM pg_replication_status"

# Standbys connected to the primary
psql -h 127.0.0.1 -p 15432 -U postgres -c "SELECT address, state, sync_mode, lag_ms FROM pg_replication_standbys"
```

---

## Part 2: Local Deployment (Without Docker)

### Prerequisites

```bash
# In the unpacked source distribution: build Lite with replication
cargo build --release --features ha-tier1
export HELIOSDB_BIN=./target/release/heliosdb-lite

# HeliosProxy is published on crates.io (or use its container image)
cargo install heliosdb-proxy
```

### Step 1: Create Data Directories and the JWT Secret

```bash
mkdir -p /tmp/heliosdb-ha/{primary,standby-sync,standby-async}

# HELIOSDB_JWT_SECRET is mandatory: the server refuses to start without it.
openssl rand -hex 32 > /tmp/heliosdb-ha/jwt.secret
chmod 600 /tmp/heliosdb-ha/jwt.secret
```

### Step 2: Start the Primary

```bash
# Terminal 1: primary (PostgreSQL 5432, replication 5433, HTTP 8080)
export HELIOSDB_JWT_SECRET="$(cat /tmp/heliosdb-ha/jwt.secret)"
$HELIOSDB_BIN start \
  --data-dir /tmp/heliosdb-ha/primary \
  --port 5432 \
  --http-port 8080 \
  --replication-role primary \
  --replication-port 5433 \
  --sync-mode sync
```

### Step 3: Start the Standbys

```bash
# Terminal 2: synchronous standby (PostgreSQL 5442, HTTP 8081)
export HELIOSDB_JWT_SECRET="$(cat /tmp/heliosdb-ha/jwt.secret)"
$HELIOSDB_BIN start \
  --data-dir /tmp/heliosdb-ha/standby-sync \
  --port 5442 \
  --http-port 8081 \
  --replication-role standby \
  --primary-host 127.0.0.1:5433 \
  --sync-mode sync
```

```bash
# Terminal 3: asynchronous standby (PostgreSQL 5452, HTTP 8082)
export HELIOSDB_JWT_SECRET="$(cat /tmp/heliosdb-ha/jwt.secret)"
$HELIOSDB_BIN start \
  --data-dir /tmp/heliosdb-ha/standby-async \
  --port 5452 \
  --http-port 8082 \
  --replication-role standby \
  --primary-host 127.0.0.1:5433 \
  --sync-mode async
```

The listen address defaults to `127.0.0.1`. Each node needs its own `--port` and
`--http-port` on a shared host. The standbys forward writes to `127.0.0.1:5432`, the
primary's PostgreSQL port.

### Step 4: Configure and Start HeliosProxy

Create `/tmp/heliosdb-ha/proxy.toml` with the configuration from
[Part 1, Step 3](#step-3-create-the-heliosproxy-configuration), changing the listener,
the admin address and the nodes:

```toml
listen_address = "127.0.0.1:6432"
admin_address = "127.0.0.1:9090"   # loopback: no admin_token needed
tr_enabled = true
tr_mode = "session"
write_timeout_secs = 30

# [pool], [load_balancer] and [health] as in Part 1, Step 3

[[nodes]]
name = "primary"
host = "127.0.0.1"
port = 5432
role = "primary"
weight = 100
enabled = true

[[nodes]]
name = "standby-sync"
host = "127.0.0.1"
port = 5442
role = "standby"
weight = 100
enabled = true

[[nodes]]
name = "standby-async"
host = "127.0.0.1"
port = 5452
role = "standby"
weight = 100
enabled = true
```

```bash
# Terminal 4: HeliosProxy (client port 6432, admin API 9090)
heliosdb-proxy --config /tmp/heliosdb-ha/proxy.toml
```

### Port Summary

| Node | PostgreSQL (local / Docker host) | HTTP (local / Docker host) | Replication |
|------|----------------------------------|----------------------------|-------------|
| Primary | 5432 / 15432 | 8080 / 18080 | 5433 (inside the network) |
| Standby, sync | 5442 / 15442 | 8081 / 18081 | - |
| Standby, async | 5452 / 15452 | 8082 / 18082 | - |
| HeliosProxy | 6432 / 16432 | admin 9090 / 19090 | - |

The commands in the rest of this tutorial use the Docker host ports. For the local
deployment, use the local ports instead.

---

## Part 3: Write Routing

Writes reach the primary in two ways.

**Through HeliosProxy.** With `read_write_split = true`, HeliosProxy sends writes,
transaction control and `SET` statements to the node configured with `role = "primary"`,
and reads to healthy nodes chosen by `read_strategy`.

**Directly to a standby.** A standby started with `--sync-mode sync` or `semi-sync`
forwards `INSERT`, `UPDATE`, `DELETE`, `CREATE`, `DROP`, `ALTER` and `TRUNCATE` to the
primary and returns the primary's result; `SELECT` runs on the standby. A standby in
`async` mode rejects writes with SQLSTATE `25006`. Forwarding has these limits:

- Only statements sent with the simple query protocol (for example from `psql`) are
  forwarded. Statements sent with the extended query protocol (prepared statements, which
  most drivers use for parameterized queries) and REST API calls to a standby are not
  forwarded. Send application writes to the primary or through HeliosProxy.
- The standby connects to the primary without a password and runs forwarded statements as
  the `heliosdb` role, not the client's role. It works only while the primary accepts
  `trust` authentication.

### Test Write Routing

```bash
# Create a table and insert a row through HeliosProxy (routed to the primary)
psql -h 127.0.0.1 -p 16432 -U postgres -c \
  "CREATE TABLE twr_test (id INTEGER PRIMARY KEY, data TEXT)"
psql -h 127.0.0.1 -p 16432 -U postgres -c \
  "INSERT INTO twr_test (id, data) VALUES (1, 'through the proxy')"

# Insert directly on the sync standby (forwarded to the primary)
psql -h 127.0.0.1 -p 15442 -U postgres -c \
  "INSERT INTO twr_test (id, data) VALUES (2, 'forwarded by the sync standby')"

# Insert directly on the async standby (rejected with SQLSTATE 25006)
psql -h 127.0.0.1 -p 15452 -U postgres -c \
  "INSERT INTO twr_test (id, data) VALUES (3, 'rejected')"

# Both accepted rows are on the primary and, once replicated, on each standby
for port in 15432 15442 15452; do
  psql -h 127.0.0.1 -p $port -U postgres -c "SELECT * FROM twr_test ORDER BY id"
done
```

---

## Part 4: Read Routing

HeliosProxy spreads reads across healthy nodes with the `read_strategy` of
`[load_balancer]`: `round_robin`, `weighted_round_robin`, `least_connections`,
`latency_based`, `random` or `power_of_two`. A read sent to an `async` standby can return
data that is slightly behind the primary.

`pg_replication_status` reports the role of the node that answers, which shows where a
read was routed. Each `psql` call below opens a new session:

```bash
for i in 1 2 3 4 5 6; do
  psql -h 127.0.0.1 -p 16432 -U postgres -X -t -A -c "SELECT role FROM pg_replication_status"
done
```

---

## Part 5: HeliosProxy Configuration and Admin API

| Key | Meaning |
|-----|---------|
| `listen_address` | Client listener (PostgreSQL protocol) |
| `admin_address` | Admin API listener; a non-loopback address requires `admin_token` |
| `tr_enabled`, `tr_mode` | Transaction Replay: `none`, `session` (default), `select` or `transaction` |
| `write_timeout_secs` | Deadline for recovery and for writes waiting for a healthy primary (default 30) |
| `[load_balancer]` | `read_strategy`, `read_write_split`, `latency_threshold_ms` |
| `[health]` | `check_interval_secs`, `check_timeout_secs`, `failure_threshold`, `success_threshold`, `check_query` |
| `[[nodes]]` | `host`, `port`, `role` (`primary`, `standby` or `replica`), `weight`, `enabled`, optional `name` |

Lite has no `pg_is_in_recovery()` function, so keep the default static topology, where
the primary is the node configured with `role = "primary"`. The
[HeliosProxy documentation](/docs/proxy/) is the full reference.

```bash
ADMIN_TOKEN="replace-with-a-random-token"   # the admin_token from proxy.toml

# Liveness (no token needed)
curl http://127.0.0.1:19090/health

# Node health and the current topology
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://127.0.0.1:19090/nodes
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://127.0.0.1:19090/topology
```

---

## Part 6: Monitoring the Cluster

### SQL Views and Statements

| Query | Where | Returns |
|-------|-------|---------|
| `SELECT * FROM pg_replication_status` | Any node | `node_id`, `role`, `sync_mode`, `is_read_only`, `current_lsn`, `standby_count`, `primary_host`, `listen_addr`, `replication_port`, `started_at` |
| `SELECT * FROM pg_replication_standbys` | Primary | One row per connected standby: `address`, `state`, `sync_mode`, `current_lsn`, `flush_lsn`, `apply_lsn`, `lag_bytes`, `lag_ms`, `connected_at`, `last_heartbeat` |
| `SELECT * FROM pg_replication_primary` | Standby | `primary_address`, `connection_state`, `primary_lsn`, `local_lsn`, `lag_bytes`, `lag_ms`, `fencing_token`, `connected_at`, `last_heartbeat` |
| `SELECT * FROM pg_replication_metrics` | Any node | One row per metric (`metric_name`, `metric_value`, `description`) |
| `SHOW TOPOLOGY` | Any node | Nodes with alias, role, addresses, health, LSN, lag |
| `SHOW CLUSTER STATUS` | Any node | This node and its connected standbys with role, LSN and lag |

```bash
# Replication lag per standby, from the primary
psql -h 127.0.0.1 -p 15432 -U postgres -c \
  "SELECT address, state, sync_mode, lag_bytes, lag_ms FROM pg_replication_standbys"

# Cluster topology
psql -h 127.0.0.1 -p 15432 -U postgres -c "SHOW TOPOLOGY"
```

### Monitoring Script

Create `monitor_cluster.sh`:

```bash
#!/usr/bin/env bash
# Print node health, roles and replication lag every 2 seconds (Docker host ports).
NODES="primary:15432:18080 standby-sync:15442:18081 standby-async:15452:18082"

while true; do
  clear
  date '+%Y-%m-%d %H:%M:%S'
  printf '%-15s %-10s %-10s %-10s\n' NODE SQL HTTP ROLE
  for entry in $NODES; do
    IFS=: read -r name pg http <<< "$entry"
    role=$(psql -h 127.0.0.1 -p "$pg" -U postgres -X -t -A \
      -c "SELECT role FROM pg_replication_status" 2>/dev/null)
    sql=$([ -n "$role" ] && echo up || echo down)
    code=$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:$http/health")
    printf '%-15s %-10s %-10s %-10s\n' "$name" "$sql" "$code" "${role:--}"
  done
  echo
  echo "Standbys seen by the primary:"
  psql -h 127.0.0.1 -p 15432 -U postgres -X \
    -c "SELECT address, state, lag_bytes, lag_ms FROM pg_replication_standbys" 2>/dev/null
  echo "HeliosProxy /health: $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:19090/health)"
  sleep 2
done
```

### Logs

```bash
docker compose -f docker-compose.ha-cluster.yml logs -f
docker compose -f docker-compose.ha-cluster.yml logs -f proxy
```

---

## Part 7: Switchover

A switchover is a planned role change between the primary and a standby.

| Statement | Status |
|-----------|--------|
| `SWITCHOVER CHECK '<node>'` | Available. Run on the primary; returns `can_proceed`, `target_healthy`, `target_lsn`, `primary_lsn`, `lag_bytes`, `warnings` and `blockers` for the target standby |
| `SET NODE ALIAS 'name' FOR '<node-id>'` | Available. Names a node so the statements above accept the alias |
| `SWITCHOVER TO '<node>'` | Planned. The statement is accepted and returns a message, but roles do not change |

```bash
# Node IDs and aliases
psql -h 127.0.0.1 -p 15432 -U postgres -c "SHOW TOPOLOGY"

# Check whether a standby could take over (use a node ID from SHOW TOPOLOGY)
psql -h 127.0.0.1 -p 15432 -U postgres -c "SWITCHOVER CHECK '<standby-node-id>'"
```

Until controlled switchover is available, a role change is a manual procedure: stop
writes to the primary, wait until `pg_replication_standbys` shows the target standby at
the primary's LSN, restart the target with `--replication-role primary`, restart the old
primary as a standby of it, and change the `role` entries in HeliosProxy's `[[nodes]]`
before restarting HeliosProxy. Rehearse the procedure, including how the remaining
standbys reconnect, in a staging environment before relying on it.

---

## Part 8: Primary Outages and HeliosProxy

Automatic failover and promotion are planned. What happens today when the primary stops:

- **Reads** continue on the healthy standbys through HeliosProxy.
- **Writes** through HeliosProxy wait for the primary. When the configured primary is
  unhealthy, HeliosProxy re-checks the primary's health state every 100 ms for up to
  `write_timeout_secs` and sends the write as soon as the primary is healthy again. If the primary does not
  return within that time, the client receives an error.
- **Open sessions** survive a backend fault with `tr_mode = "session"`: a statement that
  never reached the failed node and ran outside a transaction is re-executed; otherwise
  the client receives one error (`57P01`, or `08007` when the outcome is unknown and the
  statement may have committed). See
  [Transaction Replay](/docs/proxy/transaction-replay/) in the HeliosProxy documentation.

How long a write waits also depends on the `[health]` settings: with the values above, the
proxy marks a node unhealthy after about 15 seconds of failed checks
(`failure_threshold` x `check_interval_secs`) and healthy again after about 10 seconds of
successful checks (`success_threshold` x `check_interval_secs`).

### Workload Script

Create `workload.sh`. It writes one row per second through HeliosProxy and prints the
outcome and latency of each write (GNU `date` is required for millisecond timing):

```bash
#!/usr/bin/env bash
# Usage: ./workload.sh <seconds> [proxy port]
DURATION=${1:-60}
PORT=${2:-16432}
PSQL="psql -h 127.0.0.1 -p $PORT -U postgres -X -q -t -A"

$PSQL -c "DROP TABLE IF EXISTS ha_workload"
$PSQL -c "CREATE TABLE ha_workload (id INTEGER PRIMARY KEY, written_at TEXT)"

ok=0; failed=0; i=0; end=$((SECONDS + DURATION))
while [ "$SECONDS" -lt "$end" ]; do
  i=$((i + 1))
  start=$(date +%s%3N)
  if $PSQL -c "INSERT INTO ha_workload (id, written_at) VALUES ($i, '$(date +%T)')" >/dev/null 2>&1; then
    ok=$((ok + 1)); status=ok
  else
    failed=$((failed + 1)); status=FAILED
  fi
  echo "$(date +%T) write $i $status $(( $(date +%s%3N) - start )) ms"
  sleep 1
done
echo "writes ok=$ok failed=$failed"
```

### Test a Short Primary Outage

```bash
chmod +x workload.sh
./workload.sh 90 > /tmp/outage_test.log 2>&1 &
sleep 20
docker compose -f docker-compose.ha-cluster.yml stop primary
sleep 10
docker compose -f docker-compose.ha-cluster.yml start primary
wait
grep -E 'FAILED|[0-9]{4,} ms' /tmp/outage_test.log
tail -1 /tmp/outage_test.log
```

Writes issued during the outage show a longer latency; they complete when the primary
is back within `write_timeout_secs` and fail when it is not. Repeat the test with a
stop longer than `write_timeout_secs` to see the failure case.

---

## Part 9: Standby Outages

```bash
./workload.sh 120 > /tmp/standby_test.log 2>&1 &

# One standby down: the primary keeps receiving confirmations from the other
sleep 20
docker compose -f docker-compose.ha-cluster.yml stop standby-sync
sleep 20
docker compose -f docker-compose.ha-cluster.yml start standby-sync

# Both standbys down: the sync primary waits up to 10 seconds for a confirmation
# before each change completes
sleep 20
docker compose -f docker-compose.ha-cluster.yml stop standby-sync standby-async
sleep 30
docker compose -f docker-compose.ha-cluster.yml start standby-sync standby-async
wait
```

While both standbys are down, the log shows write latencies of 10 seconds or more; they
return to normal once a standby reconnects. Standbys retry the connection to the primary
every 5 seconds; check `pg_replication_standbys` for their state and lag.

---

## Part 10: Rolling Restart

Restart the standbys one at a time, then the primary. Writes wait during the primary
restart, up to `write_timeout_secs`.

```bash
docker compose -f docker-compose.ha-cluster.yml restart standby-async
sleep 10
docker compose -f docker-compose.ha-cluster.yml restart standby-sync
sleep 10
psql -h 127.0.0.1 -p 15432 -U postgres -c \
  "SELECT address, state, lag_ms FROM pg_replication_standbys"
docker compose -f docker-compose.ha-cluster.yml restart primary
```

---

## Quick Reference

### Common Commands

```bash
# Start the cluster (with HELIOSDB_JWT_SECRET exported, see Part 1, Step 4)
docker compose -f docker-compose.ha-cluster.yml up -d

# Stop the cluster
docker compose -f docker-compose.ha-cluster.yml down

# Connect through HeliosProxy
psql -h 127.0.0.1 -p 16432 -U postgres

# HeliosProxy liveness
curl http://127.0.0.1:19090/health
```

### Troubleshooting

| Message or symptom | Cause | What to do |
|--------------------|-------|------------|
| `--replication-role '...' requires a binary built with the ha-tier1 feature` | Binary or image built without replication | Rebuild with `ha-tier1` (Part 1, Step 1) |
| `cannot execute write operations in read-only mode (async standby)` (`25006`) | Write sent to an `async` standby | Write to the primary, through HeliosProxy, or to a `sync`/`semi-sync` standby |
| `Failed to forward query to primary` (`08006`) | The standby cannot reach the primary's PostgreSQL port, or the primary requires a password | Check `--primary-host`, `HELIOSDB_PRIMARY_PG_PORT` and the primary's authentication |
| Writes take 10 seconds or longer | `sync` or `semi-sync` primary with no connected standby | Start a standby, or use `--sync-mode async` |
| HeliosProxy refuses to start because of `admin_address` | Non-loopback admin API without `admin_token` | Set `admin_token` or bind the admin API to `127.0.0.1` |
| `NoHealthyNodes` from HeliosProxy | No healthy primary within `write_timeout_secs`, or no healthy node for a read | Check `curl .../nodes` and the node logs |

## See Also

- [High Availability](/docs/lite/features/high-availability/): replication flags, sync modes and views
- [HeliosProxy Integration Guide](/docs/lite/guides/heliosproxy_integration_guide/)
- [Docker Deployment](/docs/lite/guides/docker_deployment/)
- [HeliosProxy documentation](/docs/proxy/)