High Availability (HA)

HeliosDB Lite replicates by streaming its write-ahead log (WAL) from a primary to one or more standbys. Clients can read from any node, and a sync or semi-sync standby forwards the writes it receives to the primary, within the limits described in Write Forwarding from Standbys.

What Is Available and What Is Planned

Capability	Status
WAL streaming: primary and standby roles	Available (build with the `ha-tier1` feature)
Sync modes `async`, `semi-sync`, `sync`	Available
Write forwarding from `sync` and `semi-sync` standbys	Available, with the limits below
Replication system views, `SHOW TOPOLOGY`, `SHOW CLUSTER STATUS`, `SWITCHOVER CHECK`	Available
Controlled switchover (`SWITCHOVER TO`)	Planned: the statement is accepted but does not change roles
Automatic failover / promotion	Planned; use an external HA manager
Observer nodes for split-brain protection	Planned: the `observer` role starts no service
Multi-primary replication	Planned
Sharding	Planned
Logical replication	Planned

Building with Replication

Replication is a build-time feature of the source distribution:

cargo build --release --features ha-tier1

A binary built without it refuses to start with any --replication-role other than standalone. The source distribution marks the replication feature as experimental, and the stock Dockerfile builds without it.

Architecture

┌─────────────┐     WAL Stream    ┌─────────────┐
│   Primary   │ ───────────────→ │   Standby   │
│   (Active)  │                   │  (Passive)  │
└─────────────┘                   └─────────────┘
       ↓                                 ↓
   Read/Write                  Reads local, writes routed

Standbys connect to the primary’s replication port; the primary accepts up to 10 standbys and needs no list of them.

Starting a Cluster

export HELIOSDB_JWT_SECRET="$(openssl rand -hex 32)"

# Primary
heliosdb-lite start --data-dir /var/lib/heliosdb --listen 0.0.0.0 \
  --auth scram-sha-256 --password "$HELIOSDB_ADMIN_PASSWORD" \
  --replication-role primary --replication-port 5433 --sync-mode semi-sync

# Standby
heliosdb-lite start --data-dir /var/lib/heliosdb --listen 0.0.0.0 \
  --auth scram-sha-256 --password "$HELIOSDB_ADMIN_PASSWORD" \
  --replication-role standby --primary-host primary.example.com:5433

With password or SCRAM authentication on the primary, send writes to the primary (or through HeliosProxy): write forwarding from a standby needs trust authentication on the primary, see Write Forwarding from Standbys.

Flag	Description	Default
`--replication-role`	`standalone`, `primary`, `standby` or `observer`	`standalone`
`--replication-port`	WAL streaming port	`5433`
`--primary-host`	Primary a standby connects to (`host:port` of its replication port)	none
`--standby-hosts`	Accepted and printed at startup; standbys connect to the primary without it	none
`--observer-hosts`	Accepted; observer-based split-brain protection is planned	none
`--sync-mode`	`async`, `semi-sync` or `sync`	`async`
`--node-id`	Node UUID, generated when omitted	generated
`--replication-tls`	Encrypt WAL traffic	on when `--tls-cert` is set

Sync Modes

Mode	On the primary	On a standby
`sync`, `semi-sync`	Each change waits until one standby confirms it has received it. If no standby confirms within 10 seconds, the change completes on the primary and a warning is logged.	Forwards writes to the primary
`async`	Changes do not wait for standbys	Rejects writes (SQLSTATE `25006`)

With no standby connected, a sync or semi-sync primary waits the full 10 seconds on every change, so writes slow down until a standby reconnects.

Docker Compose

The Docker Deployment guide has a primary and standby Compose file built from the source distribution.

Write Forwarding from Standbys

A standby in sync or semi-sync mode forwards writes it receives to the primary, so a client connected to that standby can write without a second connection to the primary.

How It Works

Application → Standby → (DML/DDL forwarded) → Primary
                ↓
         (SELECT executed locally)

Behavior by Sync Mode

Sync Mode	DQL (SELECT)	DML (INSERT/UPDATE/DELETE)
sync	Execute locally on standby	Forward to primary, return result
semi-sync	Execute locally on standby	Forward to primary, return result
async	Execute locally on standby	Reject (traditional read-only)

Operations Subject to Routing

When connected to a standby in sync/semi-sync mode:

Operation	Behavior
`SELECT`	Execute locally (DQL)
`INSERT`	Forward to primary (DML)
`UPDATE`	Forward to primary (DML)
`DELETE`	Forward to primary (DML)
`CREATE`	Forward to primary (DDL)
`DROP`	Forward to primary (DDL)
`ALTER`	Forward to primary (DDL)
`TRUNCATE`	Forward to primary (DDL)

Limits

Only statements sent with the simple query protocol (for example from psql) are forwarded. Statements sent with the extended query protocol (prepared statements, which most drivers use for parameterized queries) and REST API calls to a standby are not forwarded; send application writes to the primary or through HeliosProxy.
The standby connects to the primary’s PostgreSQL port without a password and runs the forwarded statement as the heliosdb role, not the client’s role, so forwarding works only while the primary accepts trust authentication. The primary’s PostgreSQL port is taken as 5432 unless the standby’s environment sets HELIOSDB_PRIMARY_PG_PORT.

Example

-- Connect to STANDBY and execute INSERT (forwarded to primary)
INSERT INTO users VALUES (3, 'Charlie');
-- Result: INSERT 0 1 (success - executed on primary)

-- SELECT always executes locally on the connected standby
SELECT * FROM users;

For applications, HeliosProxy in front of the cluster is the simpler route: it sends writes to the primary and spreads reads across healthy nodes for every protocol; see the HA hands-on tutorial.

Monitoring

HA System Views

HeliosDB Lite has SQL system views for the HA configuration and replication metrics.

pg_replication_status

View node configuration and role:

SELECT * FROM pg_replication_status;

Column	Description
`node_id`	Unique identifier for this node
`role`	primary, standby, observer, or standalone
`sync_mode`	async, semi-sync, or sync
`is_read_only`	true on a standby
`current_lsn`	Current log sequence number
`standby_count`	Number of connected standbys (primary only)
`primary_host`	Primary this node replicates from (standby only)
`listen_addr`	Listen address
`replication_port`	WAL streaming port
`started_at`	Time the node started

pg_replication_standbys (Primary Only)

View connected standbys:

SELECT * FROM pg_replication_standbys;

Column	Description
`node_id`	Standby’s unique identifier
`address`	Standby’s connection address
`sync_mode`	Replication mode for this standby
`state`	connecting, streaming, catching_up, synced, disconnected
`current_lsn`	Standby’s current LSN position
`flush_lsn`	Flushed LSN
`apply_lsn`	Applied LSN
`lag_bytes`	Replication lag in bytes
`lag_ms`	Replication lag in milliseconds
`connected_at`	Connection timestamp
`last_heartbeat`	Last heartbeat received

pg_replication_primary (Standby Only)

View primary connection status:

SELECT * FROM pg_replication_primary;

Column	Description
`primary_node_id`	Primary’s unique identifier
`primary_address`	Primary’s address
`connection_state`	disconnected, connecting, connected, streaming, error
`primary_lsn`	Primary’s current LSN
`local_lsn`	Local LSN position
`lag_bytes`	Replication lag in bytes
`lag_ms`	Replication lag in milliseconds
`fencing_token`	Split-brain protection token
`connected_at`	Connection timestamp
`last_heartbeat`	Last heartbeat received

pg_replication_metrics

View replication metrics, one row per metric with the columns metric_name, metric_value and description:

SELECT * FROM pg_replication_metrics;

Metric	Description
`wal_writes`	Total WAL write operations
`wal_bytes_written`	Total WAL bytes written
`records_replicated`	Records sent to standbys
`bytes_replicated`	Bytes sent to standbys
`heartbeats_sent`	Heartbeats sent
`heartbeats_received`	Heartbeats received
`reconnect_count`	Number of reconnections
`current_lsn`	Current log sequence number
`standby_count`	Connected standbys
`last_wal_write_epoch`	Time of the last WAL write (Unix epoch)
`last_replication_epoch`	Time of the last replication (Unix epoch)
`ha_enabled`	Whether replication is active on this node

Monitoring Examples

-- Check if standbys are in sync
SELECT
    node_id,
    CASE
        WHEN lag_ms < 1000 THEN 'IN_SYNC'
        WHEN lag_ms < 60000 THEN 'CATCHING_UP'
        ELSE 'LAGGING'
    END as status,
    lag_ms
FROM pg_replication_standbys;

-- This node's role and position
SELECT node_id, role, current_lsn
FROM pg_replication_status;

-- All nodes known to this node
SHOW TOPOLOGY;

Failover

Automatic failover is planned. Today, promotion is performed by an external HA manager (or by an operator): stop writes to the failed primary, restart the chosen standby as --replication-role primary, and point the other standbys at it. HeliosProxy can route clients to the current primary; see the HeliosProxy Integration Guide.

Best Practices

Network: Use a dedicated replication network, and --replication-tls (on by default when --tls-cert is set)
Monitoring: Alert when lag_ms in pg_replication_standbys exceeds your threshold
Testing: Rehearse the manual promotion procedure in a staging environment
Backups: Keep taking backups; replication does not replace them