---
title: "High Availability Hands-On Tutorial"
description: "Run a HeliosDB Nano primary with two streaming-replication standbys using the published binary: configuration, monitoring, restarts and current limits."
---

This tutorial builds a three-node replication setup (one primary, two standbys) on
one host with the published `heliosdb-nano` binary, then shows how to monitor it,
what a standby accepts, what happens when a node stops, and which high-availability
functions are planned. To run the same setup in containers, see
[High Availability Cluster with Docker Compose](https://www.heliosdb.com/docs/nano/guides/ha-cluster-tutorial/).

Replication is part of the default build (the `ha-tier1` feature), so the release
binaries and the container image `ghcr.io/heliosdatabase/heliosdb-nano` include it.
No extra build flags are needed.

## What is available today

| Capability | Status |
|------------|--------|
| WAL streaming from one primary to standbys | Available |
| Sync modes `async`, `semi-sync`, `sync` | Available (see [Sync modes](#sync-modes)) |
| Reads on a standby over the PostgreSQL protocol | Available once the standby has caught up |
| Writes on a standby | Refused with SQLSTATE 25006; send writes to the primary |
| Standby catch-up after a restart | Available while the primary still holds the WAL the standby is missing |
| Replication views (`pg_replication_status`, `pg_replication_standbys`, `pg_replication_primary`, `pg_replication_metrics`) | Available |
| Shared-token authentication of the replication channel | Available (`--replication-token-file`) |
| Encryption of the replication channel | Planned; run it over loopback or a private, protected network |
| Switchover and promotion (`SWITCHOVER TO`) | Planned; the statement is refused today |
| Automatic failover and observer nodes | Planned; `--replication-role observer` starts no services |
| Transparent write forwarding from a standby to the primary | Planned |
| Adding a standby to a primary that already holds data (re-seeding) | Planned |

## How replication works

```
 clients (writes)            clients (reads)
       |                     |            |
       v                     v            v
 +-----------+        +-----------+  +-----------+
 |  primary  |        | standby 1 |  | standby 2 |
 | PG  :5432 |        | PG  :5442 |  | PG  :5452 |
 | repl:5433 |<-------| (client)  |  | (client)  |
 +-----------+        +-----------+  +-----------+
       ^                                   |
       +-----------------------------------+
         native replication port (WAL stream)
```

- The primary opens a native replication listener on its `--listen` address at
  `--replication-port` (default 5433). This port is separate from the PostgreSQL
  port and the HTTP port.
- Every committed change is written to the primary's logical WAL and streamed to
  connected standbys. A standby applies each entry and stores its applied position
  in the same write as the row, so a restart resumes from a durable position.
- On its first start the primary records a dataset history ID in its data directory
  (`NANO-PRIMARY-HISTORY.json`). On its first connection a standby binds to that
  history (`NANO-STANDBY-BINDING.json`) and afterwards refuses any other primary
  dataset. Do not edit or delete these files.
- A standby opens its PostgreSQL listener only after it has verified the primary's
  history and caught up. Its HTTP listener serves only `/`, `/health` and `/version`,
  and MySQL listener options are refused on a standby.

## Part 1: Start three nodes on one host

### Prerequisites

- The `heliosdb-nano` binary on your `PATH` (see [Installation](https://www.heliosdb.com/docs/nano/getting-started/installation/)).
- `psql` and `curl`.
- Free local ports 5432, 5433, 5442 and 5452 (PostgreSQL and replication) and 8080 to 8082 (HTTP).

### Step 1: Create a working directory and a replication token

```bash
mkdir -p ~/nano-ha && cd ~/nano-ha

# One owner-only file with exactly 32 random bytes, shared by all nodes.
head -c 32 /dev/urandom > replication.token
chmod 600 replication.token
```

The token file must be a regular file (not a symbolic link) owned by the user that
runs `heliosdb-nano`, with mode `0600` or stricter and exactly 32 bytes. When the
primary has a token, it refuses any standby that does not present the same one.

Do not create the standby data directories. A standby's data directory must not
exist before its first start: the standby creates it and binds it to the primary.
An existing directory, even an empty one, is refused.

### Step 2: Start the primary

Terminal 1:

```bash
cd ~/nano-ha
heliosdb-nano start \
  --data-dir "$PWD/primary" \
  --port 5432 \
  --http-port 8080 \
  --replication-role primary \
  --replication-port 5433 \
  --sync-mode semi-sync \
  --replication-token-file "$PWD/replication.token"
```

The startup banner lists the replication settings:

```
      - Replication role: primary
      - Replication port: 5433
      - History startup timeout: 30s
      - Sync mode: semi-sync
```

All three nodes listen on `127.0.0.1` (the default `--listen`), so the token never
leaves the host. On loopback the default `trust` authentication is accepted; for
anything beyond a local test, use `--auth scram-sha-256` with `--password-file`
(see [Security Hardening](https://www.heliosdb.com/docs/nano/guides/security_hardening/)).

### Step 3: Start two standbys

Start the standbys right after the primary, before you write any data (see
[Restarting a standby](#part-4-restart-a-standby) for why).

Terminal 2:

```bash
cd ~/nano-ha
heliosdb-nano start \
  --data-dir "$PWD/standby1" \
  --port 5442 \
  --http-port 8081 \
  --replication-role standby \
  --primary-host 127.0.0.1:5433 \
  --sync-mode semi-sync \
  --replication-token-file "$PWD/replication.token"
```

Terminal 3:

```bash
cd ~/nano-ha
heliosdb-nano start \
  --data-dir "$PWD/standby2" \
  --port 5452 \
  --http-port 8082 \
  --replication-role standby \
  --primary-host 127.0.0.1:5433 \
  --sync-mode semi-sync \
  --replication-token-file "$PWD/replication.token"
```

`--primary-host` is the primary's replication endpoint (`host:port` of
`--replication-port`), not its PostgreSQL port. Each standby must verify the
primary's history and catch up within `--replication-startup-timeout-secs`
(default 30, range 1 to 3600); otherwise it exits with an error. If a first start
fails, keep the directory it created for inspection and start again with a new,
unused `--data-dir` path.

### Step 4: Check the nodes

```bash
# Role and settings of each node
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT node_id, role, sync_mode, is_read_only, current_lsn, standby_count FROM pg_replication_status;"
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT node_id, role, sync_mode, is_read_only FROM pg_replication_status;"

# Standbys connected to the primary (run on the primary)
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT node_id, address, state, sync_mode, flush_lsn, apply_lsn, lag_bytes, lag_ms FROM pg_replication_standbys;"

# The primary connection as seen by a standby (run on a standby)
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT primary_address, connection_state, primary_lsn, local_lsn, lag_bytes FROM pg_replication_primary;"

# HTTP health of each node
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8081/health
```

`/health` answers `{"status":"ok"}`. A standby's HTTP listener starts only after
it has caught up, so a healthy standby is one that is serving reads.

## Part 2: Replicate data

Write on the primary:

```bash
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres <<'SQL'
CREATE TABLE orders (id INT PRIMARY KEY, item TEXT, qty INT);
INSERT INTO orders VALUES (1, 'widget', 3), (2, 'gadget', 5);
SQL
```

Read the same rows on each standby:

```bash
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT * FROM orders ORDER BY id;"
psql -h 127.0.0.1 -p 5452 -U postgres -d postgres -c "SELECT * FROM orders ORDER BY id;"
```

Table definitions and rows both replicate. Compare `current_lsn` in
`pg_replication_status` on the primary with `apply_lsn` in `pg_replication_standbys`
to see how far each standby has applied.

### Sync modes

`--sync-mode` on the primary decides whether a commit waits for a standby:

| Mode | Commit on the primary |
|------|-----------------------|
| `async` (default) | Returns as soon as the primary has written it. |
| `semi-sync` | Waits until at least one connected standby has received the entry, for up to 10 seconds. |
| `sync` | Same wait as `semi-sync` in the current release: at least one standby must have received the entry, for up to 10 seconds. |

When no standby is connected, no wait happens. When the 10-second wait runs out,
the primary logs a warning and the commit stands: it is not rolled back. Use
`semi-sync` or `sync` to narrow the window in which a committed change exists only
on the primary; do not read either mode as a guarantee of zero data loss.

## Part 3: Writes on a standby are refused

```bash
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "INSERT INTO orders VALUES (3, 'gizmo', 1);"
```

```
ERROR:  transparent forwarding is disabled for a history-guarded standby
HINT:  Connect directly to the primary for write operations.
```

The SQLSTATE is `25006` (read-only transaction). Send all writes to the primary.
This refusal covers INSERT, UPDATE, DELETE, CREATE, DROP, ALTER and TRUNCATE sent
as simple queries; it is not a complete read-only security boundary, so do not give
write access to applications through a standby. Forwarding writes from a standby
to the primary is planned.

To give applications one endpoint that sends writes to the primary and reads to
standbys, put [HeliosProxy](/docs/proxy/) in front of the nodes; see the
[HeliosProxy configuration reference](/docs/proxy/configuration/).

## Part 4: Restart a standby

Stop standby 2 (Ctrl+C in terminal 3), then start it again with the same command.
Its data directory now carries a binding, so the standby reconnects, verifies the
same primary history, applies the entries it missed and only then opens its
PostgreSQL listener.

A standby can catch up only while the primary still holds the WAL above the
standby's position. The primary truncates its logical WAL at a checkpoint, which it
takes on a write once 1,000 entries or 30 seconds have passed since the previous
checkpoint, and at a clean shutdown. The keys are `[storage]
wal_checkpoint_interval_entries` (default 1000) and `wal_checkpoint_interval_secs`
(default 30); larger values keep more WAL for returning standbys and make crash
recovery replay more entries.

If the primary has checkpointed past the standby's position, the standby is refused
and stops:

```
full resync required: primary history was checkpointed through LSN <checkpoint>; the standby (at LSN <position>) must be rebuilt
```

The same applies to a new standby added to a primary that already holds data.
Re-seeding a standby from a running primary is planned. Until then, plan standby
maintenance so that a standby returns before the primary checkpoints past it, and
keep a [backup](https://www.heliosdb.com/docs/nano/guides/tutorials/backup_restore_tutorial/) of the primary.

## Part 5: When the primary stops

Stop the primary (Ctrl+C in terminal 1) and observe:

```bash
# The standbys keep serving reads
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT count(*) FROM orders;"

# Writes have nowhere to go until the primary is back
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT 1;"   # connection refused
```

Each standby keeps retrying the connection, first after 5 seconds and then with a
delay that doubles up to 60 seconds. Start the
primary again with its original command: the standbys reconnect and resume
streaming. A clean shutdown checkpoints the primary at its last entry, which a
standby that was caught up already holds, so it resumes without a resync.

Promotion of a standby is planned. Today:

- `SWITCHOVER TO '<node>'` checks that it runs on the primary and that the target is a
  connected standby, and then returns `Switchover refused: runtime switchover/promotion
  is unsupported ...`. No role changes.
- `SWITCHOVER CHECK '<node>'` reports the target's health with `can_proceed = false`.
- Starting a standby's data directory with `--replication-role primary` is refused
  (`primary directory carries a standby binding; implicit promotion is refused`).

Plan for a primary outage as an outage for writes: restart the primary on its data
directory, or restore it from a backup.

## Part 6: Stop the setup

Press Ctrl+C in each terminal (standbys first, then the primary). To start over,
delete `~/nano-ha/primary`, `~/nano-ha/standby1` and `~/nano-ha/standby2`; a new
primary gets a new history, and the standbys need new, absent data directories.

## Configuration file

Every flag has a key in the `[replication]` section of the configuration file
passed with `--config`. Flags given on the command line override the file.

```toml
# standby.toml
[replication]
role = "standby"                 # standalone (default), primary, standby, observer
primary_host = "127.0.0.1:5433"  # primary's replication endpoint (standby only)
replication_port = 5433          # replication listener port (primary)
sync_mode = "semi-sync"          # async (default), semi-sync, sync
startup_timeout_secs = 30        # 1 to 3600
token_file = "/etc/heliosdb/replication.token"   # 32 bytes, mode 0600
# node_id = "6f1c2e3a-..."       # UUID; generated at startup when absent
```

```bash
heliosdb-nano start --data-dir "$PWD/standby1" --port 5442 --http-port 8081 --config standby.toml
```

## Flag reference

| Flag | Meaning |
|------|---------|
| `--replication-role` | `standalone` (default), `primary`, `standby` or `observer` |
| `--replication-port` | Replication listener port on the `--listen` address (default 5433) |
| `--primary-host` | `host:port` of the primary's replication listener; required for a standby |
| `--sync-mode` | `async` (default), `semi-sync` or `sync` |
| `--replication-token-file` | Path of the shared 32-byte token file |
| `--allow-insecure-replication-remote` | Allow the token on a non-loopback replication listener (plaintext); use only on a private, protected network |
| `--replication-startup-timeout-secs` | Time a standby has to verify the primary and catch up (default 30) |
| `--node-id` | Node UUID; generated at startup when absent |
| `--standby-hosts`, `--observer-hosts` | Accepted and validated; not used by the current release |
| `--http-port` | HTTP listener port (default 8080; `0` disables it) |

## Troubleshooting

| Message | Cause and fix |
|---------|---------------|
| `standby role requires --primary-host or [replication] primary_host` | Add `--primary-host host:port`. |
| `existing receiver directory has no binding: ...; preserve it and use a new data directory` | The standby's `--data-dir` already existed. Start the standby with a path that does not exist yet. |
| `Standby did not verify primary dataset history within 30s (replication.startup_timeout_secs)` | The primary is not reachable at `--primary-host`, or the token differs. Check the endpoint and the token file, then use a new data directory. |
| `full resync required: ...` | The primary no longer holds the WAL this standby needs. See [Part 4](#part-4-restart-a-standby). |
| `native replication token is a plaintext bearer; bind --listen to a loopback address, ...` | The primary listens on a non-loopback address with a token. Use loopback, or add `--allow-insecure-replication-remote` on a private, protected network. |
| `replication token must be an owned, nonsymlink, single-link regular file, mode 0600 or stricter, containing exactly 32 raw bytes` | Recreate the token file with `head -c 32 /dev/urandom` and `chmod 600`, owned by the user running the server. |
| `Cannot bind native replication listener ...` | `--port` and `--replication-port` collide, or the port is in use. Give each its own port. |
| `history-guarded standby does not support MySQL listeners; ...` | Remove `--mysql` / `--mysql-socket` from the standby. |
| `transparent forwarding is disabled for a history-guarded standby` | A write reached a standby. Connect to the primary. |

## Next steps

- [High Availability Cluster with Docker Compose](https://www.heliosdb.com/docs/nano/guides/ha-cluster-tutorial/): the same
  topology with the published container image.
- [High Availability Checkpoints](https://www.heliosdb.com/docs/nano/guides/ha_interactive_tutorial/): short exercises to run
  against this setup.
- [High Availability](https://www.heliosdb.com/docs/nano/features/high-availability/): feature overview.
- [Backup and Restore](https://www.heliosdb.com/docs/nano/guides/tutorials/backup_restore_tutorial/): `heliosdb-nano dump` and `restore`.