High Availability Hands-On Tutorial

This tutorial builds a three-node replication setup (one primary, two standbys) on one host with the published heliosdb-nano binary, then shows how to monitor it, what a standby accepts, what happens when a node stops, and which high-availability functions are planned. To run the same setup in containers, see High Availability Cluster with Docker Compose.

Replication is part of the default build (the ha-tier1 feature), so the release binaries and the container image ghcr.io/heliosdatabase/heliosdb-nano include it. No extra build flags are needed.

What is available today

Capability	Status
WAL streaming from one primary to standbys	Available
Sync modes `async`, `semi-sync`, `sync`	Available (see Sync modes)
Reads on a standby over the PostgreSQL protocol	Available once the standby has caught up
Writes on a standby	Refused with SQLSTATE 25006; send writes to the primary
Standby catch-up after a restart	Available while the primary still holds the WAL the standby is missing
Replication views (`pg_replication_status`, `pg_replication_standbys`, `pg_replication_primary`, `pg_replication_metrics`)	Available
Shared-token authentication of the replication channel	Available (`--replication-token-file`)
Encryption of the replication channel	Planned; run it over loopback or a private, protected network
Switchover and promotion (`SWITCHOVER TO`)	Planned; the statement is refused today
Automatic failover and observer nodes	Planned; `--replication-role observer` starts no services
Transparent write forwarding from a standby to the primary	Planned
Adding a standby to a primary that already holds data (re-seeding)	Planned

How replication works

 clients (writes)            clients (reads)
       |                     |            |
       v                     v            v
 +-----------+        +-----------+  +-----------+
 |  primary  |        | standby 1 |  | standby 2 |
 | PG  :5432 |        | PG  :5442 |  | PG  :5452 |
 | repl:5433 |<-------| (client)  |  | (client)  |
 +-----------+        +-----------+  +-----------+
       ^                                   |
       +-----------------------------------+
         native replication port (WAL stream)

The primary opens a native replication listener on its --listen address at --replication-port (default 5433). This port is separate from the PostgreSQL port and the HTTP port.
Every committed change is written to the primary’s logical WAL and streamed to connected standbys. A standby applies each entry and stores its applied position in the same write as the row, so a restart resumes from a durable position.
On its first start the primary records a dataset history ID in its data directory (NANO-PRIMARY-HISTORY.json). On its first connection a standby binds to that history (NANO-STANDBY-BINDING.json) and afterwards refuses any other primary dataset. Do not edit or delete these files.
A standby opens its PostgreSQL listener only after it has verified the primary’s history and caught up. Its HTTP listener serves only /, /health and /version, and MySQL listener options are refused on a standby.

Part 1: Start three nodes on one host

Prerequisites

The heliosdb-nano binary on your PATH (see Installation).
psql and curl.
Free local ports 5432, 5433, 5442 and 5452 (PostgreSQL and replication) and 8080 to 8082 (HTTP).

Step 1: Create a working directory and a replication token

mkdir -p ~/nano-ha && cd ~/nano-ha

# One owner-only file with exactly 32 random bytes, shared by all nodes.
head -c 32 /dev/urandom > replication.token
chmod 600 replication.token

The token file must be a regular file (not a symbolic link) owned by the user that runs heliosdb-nano, with mode 0600 or stricter and exactly 32 bytes. When the primary has a token, it refuses any standby that does not present the same one.

Do not create the standby data directories. A standby’s data directory must not exist before its first start: the standby creates it and binds it to the primary. An existing directory, even an empty one, is refused.

Step 2: Start the primary

Terminal 1:

cd ~/nano-ha
heliosdb-nano start \
  --data-dir "$PWD/primary" \
  --port 5432 \
  --http-port 8080 \
  --replication-role primary \
  --replication-port 5433 \
  --sync-mode semi-sync \
  --replication-token-file "$PWD/replication.token"

The startup banner lists the replication settings:

      - Replication role: primary
      - Replication port: 5433
      - History startup timeout: 30s
      - Sync mode: semi-sync

All three nodes listen on 127.0.0.1 (the default --listen), so the token never leaves the host. On loopback the default trust authentication is accepted; for anything beyond a local test, use --auth scram-sha-256 with --password-file (see Security Hardening).

Step 3: Start two standbys

Start the standbys right after the primary, before you write any data (see Restarting a standby for why).

Terminal 2:

cd ~/nano-ha
heliosdb-nano start \
  --data-dir "$PWD/standby1" \
  --port 5442 \
  --http-port 8081 \
  --replication-role standby \
  --primary-host 127.0.0.1:5433 \
  --sync-mode semi-sync \
  --replication-token-file "$PWD/replication.token"

Terminal 3:

cd ~/nano-ha
heliosdb-nano start \
  --data-dir "$PWD/standby2" \
  --port 5452 \
  --http-port 8082 \
  --replication-role standby \
  --primary-host 127.0.0.1:5433 \
  --sync-mode semi-sync \
  --replication-token-file "$PWD/replication.token"

--primary-host is the primary’s replication endpoint (host:port of --replication-port), not its PostgreSQL port. Each standby must verify the primary’s history and catch up within --replication-startup-timeout-secs (default 30, range 1 to 3600); otherwise it exits with an error. If a first start fails, keep the directory it created for inspection and start again with a new, unused --data-dir path.

Step 4: Check the nodes

# Role and settings of each node
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT node_id, role, sync_mode, is_read_only, current_lsn, standby_count FROM pg_replication_status;"
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT node_id, role, sync_mode, is_read_only FROM pg_replication_status;"

# Standbys connected to the primary (run on the primary)
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT node_id, address, state, sync_mode, flush_lsn, apply_lsn, lag_bytes, lag_ms FROM pg_replication_standbys;"

# The primary connection as seen by a standby (run on a standby)
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT primary_address, connection_state, primary_lsn, local_lsn, lag_bytes FROM pg_replication_primary;"

# HTTP health of each node
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8081/health

/health answers {"status":"ok"}. A standby’s HTTP listener starts only after it has caught up, so a healthy standby is one that is serving reads.

Part 2: Replicate data

Write on the primary:

psql -h 127.0.0.1 -p 5432 -U postgres -d postgres <<'SQL'
CREATE TABLE orders (id INT PRIMARY KEY, item TEXT, qty INT);
INSERT INTO orders VALUES (1, 'widget', 3), (2, 'gadget', 5);
SQL

Read the same rows on each standby:

psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT * FROM orders ORDER BY id;"
psql -h 127.0.0.1 -p 5452 -U postgres -d postgres -c "SELECT * FROM orders ORDER BY id;"

Table definitions and rows both replicate. Compare current_lsn in pg_replication_status on the primary with apply_lsn in pg_replication_standbys to see how far each standby has applied.

Sync modes

--sync-mode on the primary decides whether a commit waits for a standby:

Mode	Commit on the primary
`async` (default)	Returns as soon as the primary has written it.
`semi-sync`	Waits until at least one connected standby has received the entry, for up to 10 seconds.
`sync`	Same wait as `semi-sync` in the current release: at least one standby must have received the entry, for up to 10 seconds.

When no standby is connected, no wait happens. When the 10-second wait runs out, the primary logs a warning and the commit stands: it is not rolled back. Use semi-sync or sync to narrow the window in which a committed change exists only on the primary; do not read either mode as a guarantee of zero data loss.

Part 3: Writes on a standby are refused

psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "INSERT INTO orders VALUES (3, 'gizmo', 1);"

ERROR:  transparent forwarding is disabled for a history-guarded standby
HINT:  Connect directly to the primary for write operations.

The SQLSTATE is 25006 (read-only transaction). Send all writes to the primary. This refusal covers INSERT, UPDATE, DELETE, CREATE, DROP, ALTER and TRUNCATE sent as simple queries; it is not a complete read-only security boundary, so do not give write access to applications through a standby. Forwarding writes from a standby to the primary is planned.

To give applications one endpoint that sends writes to the primary and reads to standbys, put HeliosProxy in front of the nodes; see the HeliosProxy configuration reference.

Part 4: Restart a standby

Stop standby 2 (Ctrl+C in terminal 3), then start it again with the same command. Its data directory now carries a binding, so the standby reconnects, verifies the same primary history, applies the entries it missed and only then opens its PostgreSQL listener.

A standby can catch up only while the primary still holds the WAL above the standby’s position. The primary truncates its logical WAL at a checkpoint, which it takes on a write once 1,000 entries or 30 seconds have passed since the previous checkpoint, and at a clean shutdown. The keys are [storage] wal_checkpoint_interval_entries (default 1000) and wal_checkpoint_interval_secs (default 30); larger values keep more WAL for returning standbys and make crash recovery replay more entries.

If the primary has checkpointed past the standby’s position, the standby is refused and stops:

full resync required: primary history was checkpointed through LSN <checkpoint>; the standby (at LSN <position>) must be rebuilt

The same applies to a new standby added to a primary that already holds data. Re-seeding a standby from a running primary is planned. Until then, plan standby maintenance so that a standby returns before the primary checkpoints past it, and keep a backup of the primary.

Part 5: When the primary stops

Stop the primary (Ctrl+C in terminal 1) and observe:

# The standbys keep serving reads
psql -h 127.0.0.1 -p 5442 -U postgres -d postgres -c "SELECT count(*) FROM orders;"

# Writes have nowhere to go until the primary is back
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT 1;"   # connection refused

Each standby keeps retrying the connection, first after 5 seconds and then with a delay that doubles up to 60 seconds. Start the primary again with its original command: the standbys reconnect and resume streaming. A clean shutdown checkpoints the primary at its last entry, which a standby that was caught up already holds, so it resumes without a resync.

Promotion of a standby is planned. Today:

SWITCHOVER TO '<node>' checks that it runs on the primary and that the target is a connected standby, and then returns Switchover refused: runtime switchover/promotion is unsupported .... No role changes.
SWITCHOVER CHECK '<node>' reports the target’s health with can_proceed = false.
Starting a standby’s data directory with --replication-role primary is refused (primary directory carries a standby binding; implicit promotion is refused).

Plan for a primary outage as an outage for writes: restart the primary on its data directory, or restore it from a backup.

Part 6: Stop the setup

Press Ctrl+C in each terminal (standbys first, then the primary). To start over, delete ~/nano-ha/primary, ~/nano-ha/standby1 and ~/nano-ha/standby2; a new primary gets a new history, and the standbys need new, absent data directories.

Configuration file

Every flag has a key in the [replication] section of the configuration file passed with --config. Flags given on the command line override the file.

[replication]
role = "standby"                 # standalone (default), primary, standby, observer
primary_host = "127.0.0.1:5433"  # primary's replication endpoint (standby only)
replication_port = 5433          # replication listener port (primary)
sync_mode = "semi-sync"          # async (default), semi-sync, sync
startup_timeout_secs = 30        # 1 to 3600
token_file = "/etc/heliosdb/replication.token"   # 32 bytes, mode 0600
# node_id = "6f1c2e3a-..."       # UUID; generated at startup when absent

heliosdb-nano start --data-dir "$PWD/standby1" --port 5442 --http-port 8081 --config standby.toml

Flag reference

Flag	Meaning
`--replication-role`	`standalone` (default), `primary`, `standby` or `observer`
`--replication-port`	Replication listener port on the `--listen` address (default 5433)
`--primary-host`	`host:port` of the primary’s replication listener; required for a standby
`--sync-mode`	`async` (default), `semi-sync` or `sync`
`--replication-token-file`	Path of the shared 32-byte token file
`--allow-insecure-replication-remote`	Allow the token on a non-loopback replication listener (plaintext); use only on a private, protected network
`--replication-startup-timeout-secs`	Time a standby has to verify the primary and catch up (default 30)
`--node-id`	Node UUID; generated at startup when absent
`--standby-hosts`, `--observer-hosts`	Accepted and validated; not used by the current release
`--http-port`	HTTP listener port (default 8080; `0` disables it)

Troubleshooting

Message	Cause and fix
`standby role requires --primary-host or [replication] primary_host`	Add `--primary-host host:port`.
`existing receiver directory has no binding: ...; preserve it and use a new data directory`	The standby’s `--data-dir` already existed. Start the standby with a path that does not exist yet.
`Standby did not verify primary dataset history within 30s (replication.startup_timeout_secs)`	The primary is not reachable at `--primary-host`, or the token differs. Check the endpoint and the token file, then use a new data directory.
`full resync required: ...`	The primary no longer holds the WAL this standby needs. See Part 4.
`native replication token is a plaintext bearer; bind --listen to a loopback address, ...`	The primary listens on a non-loopback address with a token. Use loopback, or add `--allow-insecure-replication-remote` on a private, protected network.
`replication token must be an owned, nonsymlink, single-link regular file, mode 0600 or stricter, containing exactly 32 raw bytes`	Recreate the token file with `head -c 32 /dev/urandom` and `chmod 600`, owned by the user running the server.
`Cannot bind native replication listener ...`	`--port` and `--replication-port` collide, or the port is in use. Give each its own port.
`history-guarded standby does not support MySQL listeners; ...`	Remove `--mysql` / `--mysql-socket` from the standby.
`transparent forwarding is disabled for a history-guarded standby`	A write reached a standby. Connect to the primary.

Next steps

High Availability Cluster with Docker Compose: the same topology with the published container image.
High Availability Checkpoints: short exercises to run against this setup.
High Availability: feature overview.
Backup and Restore: heliosdb-nano dump and restore.