High Availability (HA)

Overview

HeliosDB Nano organises high availability into three Cargo features. Tier 1 (warm standby) is in the default build: a primary streams its WAL to read-only standbys, configured with start flags. Tier 2 (branch-based active-active with vector-clock conflict resolution) and Tier 3 (consistent-hash sharding) are opt-in features built from source. Promotion of a standby and automatic failover are planned. For connection routing at the wire layer, put the standalone HeliosProxy in front of the nodes.

For a step-by-step setup, see the High Availability Hands-On Tutorial and High Availability Cluster with Docker Compose.

---

Tier Matrix

Tier	Cargo feature	Default?	Architecture	Conflict resolution	Best for
Tier 1	ha-tier1	yes	Warm standby (WAL streaming)	N/A (single primary)	Read replicas, standby copies
Tier 2	ha-tier2	no	Branch-based active-active	Vector clock + field-level merge	Multi-region writes, edge sync
Tier 3	ha-tier3	no	Consistent hash ring sharding	Per-shard	Horizontal scaling beyond one host

Optional add-ons

Feature	Adds	Implies
ha-dedup	Content-addressed deduplication across nodes	—
ha-ab-testing	Branch-based experiment routing	—
ha-branch-replication	Selective branch sync to remote servers	ha-tier2
ha-full	Bundle of all of the above	ha-tier1+2+3 + add-ons

Build recipes

# Default — Tier 1 only (warm standby + DR)
cargo build --release

# Add Tier 2 (multi-primary)
cargo build --release --features ha-tier2

# Full HA stack
cargo build --release --features ha-full

[dependencies]
heliosdb-nano = { version = "4", features = ["ha-tier2"] }

---

Tier 1: Warm Standby

Active-passive replication: one primary accepts writes and streams its WAL to standbys that serve reads.

Architecture

+-------------+     WAL stream     +-------------+
|   Primary   | -----------------> |   Standby   |
|  (writes)   |   replication port |   (reads)   |
+-------------+                    +-------------+

Availability

Capability	Status
WAL streaming to one or more standbys	Available in the default build
Sync modes async, semi-sync, sync	Available
Standby reads over the PostgreSQL protocol	Available after catch-up
Standby catch-up after a restart	Available while the primary holds the missing WAL
Dataset history guard (a standby never attaches to another primary’s data)	Available
Shared-token authentication (--replication-token-file)	Available
Writes on a standby	Refused (SQLSTATE 25006); forwarding to the primary is planned
Switchover and promotion	Planned; SWITCHOVER TO is refused
Automatic failover, observer quorum and split-brain fencing	Planned; --replication-role observer starts no services
Encrypted replication channel	Planned
Re-seeding a standby from a primary that already holds data	Planned

The library contains failover, split-brain and switchover components (FailoverWatcher, SplitBrainProtector, SwitchoverCoordinator), but the start command does not use them, and they are not a supported promotion path.

Sync Modes

--sync-mode on the primary decides whether a commit waits for a standby.

Mode	Commit on the primary
async (default)	Returns once the primary has written it
semi-sync	Waits until at least one connected standby has received the entry, up to 10 seconds
sync	Same wait as semi-sync in the current release

With no standby connected there is no wait. When the wait times out, the primary logs a warning and the commit stands.

---

Tier 2: Multi-Primary

Active-active replication with conflict resolution.

Architecture

┌─────────────┐   Branch Sync   ┌─────────────┐
│  Region A   │ ←─────────────→ │  Region B   │
│  (Primary)  │                 │  (Primary)  │
└─────────────┘                 └─────────────┘
    ↓     ↓                       ↓     ↓
  Writes Reads                  Writes Reads

Components

Component	Description
MultiPrimarySyncManager	Coordinates multi-region sync
ConflictMergeEngine	Resolves write conflicts
RegionCoordinator	Manages region topology

Conflict Resolution Strategies

Strategy	Description	Use Case
Last-Write-Wins	Timestamp-based	Simple, no conflicts visible
Branch-Wins	Prefer local changes	Low-latency local writes
Merge	Combine changes	Collaborative editing
Custom	User-defined logic	Complex business rules

Configuration

use heliosdb_nano::replication::{
    MultiPrimarySyncManager,
    ConflictResolution,
};

let sync = MultiPrimarySyncManager::new()
    .add_region("us-east", "us-east.example.com:5432")
    .add_region("eu-west", "eu-west.example.com:5432")
    .conflict_resolution(ConflictResolution::LastWriteWins)
    .build();

Branch-Based Replication

Multi-primary uses HeliosDB Nano’s branching for conflict-free merges:

-- Each region maintains its own branch
-- Sync merges branches across regions

-- Region A writes
INSERT INTO orders (id, total) VALUES (1, 100);

-- Region B writes (concurrent)
INSERT INTO orders (id, total) VALUES (2, 200);

-- After sync: both rows present in all regions

---

Tier 3: Sharding

Horizontal scaling with consistent hashing.

Architecture

                    ┌─────────────┐
                    │   Router    │
                    └──────┬──────┘
           ┌───────────────┼───────────────┐
           ↓               ↓               ↓
    ┌──────────┐    ┌──────────┐    ┌──────────┐
    │ Shard 1  │    │ Shard 2  │    │ Shard 3  │
    │  (0-33%) │    │ (34-66%) │    │ (67-100%)│
    └──────────┘    └──────────┘    └──────────┘

Components

Component	Description
HashRing	Consistent hashing for key distribution
ShardRouter	Routes queries to correct shard
ReshardManager	Online resharding with minimal downtime
VectorPartitioner	Special partitioning for vector data

Sharding Strategies

Strategy	Description	Best For
Hash	Consistent hash of shard key	Even distribution
Range	Key ranges per shard	Time-series data
Geographic	Location-based routing	Multi-region
Vector	Centroid-based partitioning	Vector search

Configuration

use heliosdb_nano::replication::{HashRing, ShardRouter};

let ring = HashRing::new()
    .add_node("shard1.example.com:5432", 100)  // weight: 100
    .add_node("shard2.example.com:5432", 100)
    .add_node("shard3.example.com:5432", 100)
    .build();

let router = ShardRouter::new(ring)
    .shard_key("tenant_id")  // Shard by tenant
    .build();

Vector Partitioning

Special support for vector workloads:

use heliosdb_nano::replication::{VectorPartitioner, CentroidManager};

let partitioner = VectorPartitioner::new()
    .dimensions(768)
    .num_centroids(16)  // 16 partitions based on vector similarity
    .build();

// Vectors routed to shard containing nearest centroid

Resharding

Online resharding without downtime:

use heliosdb_nano::replication::ReshardManager;

let reshard = ReshardManager::new(ring)
    .target_shards(6)  // Scale from 3 to 6 shards
    .parallel_streams(4)
    .build();

reshard.execute().await?;  // Non-blocking migration

---

Logical Replication

For selective table replication:

use heliosdb_nano::replication::{
    LogicalReplicationPipeline,
    TableFilter,
    ColumnMapping,
};

let pipeline = LogicalReplicationPipeline::new()
    .source("source.example.com:5432")
    .destination("dest.example.com:5432")
    .table_filter(TableFilter::include(&["users", "orders"]))
    .column_mapping(ColumnMapping::new()
        .rename("old_name", "new_name")
        .exclude("sensitive_column"))
    .build();

pipeline.start().await?;

---

CLI Options

Start HeliosDB Nano in HA (Tier 1 warm standby) mode with the start subcommand’s replication flags:

# Token shared by all nodes: 32 random bytes, mode 0600, owned by the server user
head -c 32 /dev/urandom > /etc/heliosdb/replication.token
chmod 600 /etc/heliosdb/replication.token

# Primary: replication listener on the --listen address at --replication-port
heliosdb-nano start --data-dir /var/lib/heliosdb/primary --listen 10.0.0.10 \
    --auth scram-sha-256 --password-file /etc/heliosdb/password \
    --replication-role primary \
    --replication-port 5433 \
    --sync-mode semi-sync \
    --replication-token-file /etc/heliosdb/replication.token \
    --allow-insecure-replication-remote

# Standby: the data directory must not exist before the first start
heliosdb-nano start --data-dir /var/lib/heliosdb/standby --listen 10.0.0.11 \
    --auth scram-sha-256 --password-file /etc/heliosdb/password \
    --replication-role standby \
    --primary-host 10.0.0.10:5433 \
    --replication-token-file /etc/heliosdb/replication.token

The replication channel is not encrypted. A primary with a token on a non-loopback address starts only with --allow-insecure-replication-remote; run that only on a private network or through a protected tunnel (for example WireGuard).

Flag	Default	Description
--replication-role	standalone	standalone, primary, standby or observer
--replication-port	5433	Port used for WAL streaming
--primary-host	—	host:port of the primary’s replication listener (standby)
--sync-mode	async	async, semi-sync or sync
--replication-token-file	—	Shared 32-byte token file (mode 0600)
--allow-insecure-replication-remote	off	Allow the token on a non-loopback listener (plaintext)
--replication-startup-timeout-secs	30	Time a standby has to verify the primary and catch up
--node-id	auto	Node UUID; generated if omitted
--standby-hosts, --observer-hosts	—	Accepted and validated; not used by the current release

The same settings can go in the [replication] section of the configuration file (role, replication_port, primary_host, sync_mode, startup_timeout_secs, token_file, node_id).

Tier 2 (multi-primary) and Tier 3 (sharding) are configured through the Rust API (see their Configuration sections above), not through start command-line flags.

---

Docker Support

The official image ghcr.io/heliosdatabase/heliosdb-nano includes Tier 1. A standby container needs one adjustment: the image entrypoint creates the data directory before starting the server, and a standby refuses a data directory that already exists, so standbys run the binary directly. The complete Compose file is in High Availability Cluster with Docker Compose.

---

Writes on a Standby

A standby refuses write statements in every sync mode:

ERROR:  transparent forwarding is disabled for a history-guarded standby
HINT:  Connect directly to the primary for write operations.

The SQLSTATE is 25006. Send writes to the primary. The check covers INSERT, UPDATE, DELETE, CREATE, DROP, ALTER and TRUNCATE sent as simple queries; it is not a complete read-only security boundary. Forwarding writes from a standby to the primary (transparent write routing) is planned.

---

Monitoring

HA System Views

HeliosDB Nano provides SQL system views for monitoring HA configuration and replication metrics.

pg_replication_status

View node configuration and role:

SELECT * FROM pg_replication_status;

Column	Description
node_id	Unique identifier for this node
role	primary, standby, observer, or standalone
sync_mode	async, semi-sync, or sync
is_read_only	true on a standby
current_lsn	Current log sequence number
standby_count	Number of connected standbys (primary only)
primary_host	Configured primary endpoint (standby only)
listen_addr	Listen address
replication_port	Replication listener port
started_at	Time the node started

pg_replication_standbys (Primary Only)

View connected standbys:

SELECT * FROM pg_replication_standbys;

Column	Description
node_id	Standby’s unique identifier
address	Standby’s connection address
sync_mode	Replication mode for this standby
state	connecting, streaming, catching_up, synced, disconnected
current_lsn	Standby’s current LSN position
flush_lsn	Flushed LSN
apply_lsn	Applied LSN
lag_bytes	Replication lag in bytes
lag_ms	Replication lag in milliseconds
connected_at	Connection timestamp
last_heartbeat	Last heartbeat received

pg_replication_primary (Standby Only)

View primary connection status:

SELECT * FROM pg_replication_primary;

Column	Description
primary_node_id	Primary’s unique identifier
primary_address	Primary’s address
connection_state	disconnected, connecting, connected, streaming, error
primary_lsn	Primary’s current LSN
local_lsn	Local LSN position
lag_bytes	Replication lag in bytes
lag_ms	Replication lag in milliseconds
fencing_token	Token issued by the primary in the replication handshake
connected_at	Connection timestamp
last_heartbeat	Last heartbeat received

pg_replication_metrics

View performance metrics:

SELECT * FROM pg_replication_metrics;

One row per metric:

Column	Description
metric_name	Name of the metric
metric_value	Current value
description	What the metric counts

Monitoring Examples

-- Check if standbys are in sync
SELECT
    node_id,
    CASE
        WHEN lag_ms < 1000 THEN 'IN_SYNC'
        WHEN lag_ms < 60000 THEN 'CATCHING_UP'
        ELSE 'LAGGING'
    END as status,
    lag_ms
FROM pg_replication_standbys;

-- View all nodes in cluster
SELECT node_id, role, current_lsn
FROM pg_replication_status;

---

Best Practices

1. Network: Run replication over loopback, a private network or a protected tunnel; the channel is not encrypted.
2. Token: Always set --replication-token-file so only nodes with the token can attach as standbys.
3. Monitoring: Alert on lag_bytes / lag_ms in pg_replication_standbys and on standbys missing from it.
4. Standby maintenance: Bring a stopped standby back before the primary checkpoints past its position (by default 1,000 entries or 30 seconds of writes); otherwise it needs a rebuild.
5. Backups: Keep taking backups with heliosdb-nano dump; a standby is not a backup and cannot be promoted yet.

---

HeliosProxy: Wire-Level Routing

For PostgreSQL-wire connection routing and read/write splitting between nodes, deploy the standalone HeliosProxy in front of the nodes:

• Repository: github.com/HeliosDatabase/HeliosDB-Proxy
• Documentation: HeliosProxy
• Topology: sits between clients and the Nano nodes, speaks the PostgreSQL wire protocol, sends writes to the primary and reads to standbys.

Because Nano standbys cannot be promoted yet, a primary outage is an outage for writes even behind the proxy.

---

See Also

• Configuration
• Deployment Modes
• Multi-Tenancy
• BACKUP_RESTORE_TUTORIAL — keep PITR running alongside HA
• FIPS_COMPLIANCE_TUTORIAL — HA in regulated deployments