---
title: High Availability (HA)
features: ha-tier1 (default), ha-tier2, ha-tier3, ha-dedup, ha-ab-testing, ha-branch-replication, ha-full
---

# High Availability (HA)

## Overview

HeliosDB Nano organises high availability into three Cargo features. **Tier 1
(warm standby)** is in the default build: a primary streams its WAL to read-only
standbys, configured with `start` flags. **Tier 2** (branch-based active-active with
vector-clock conflict resolution) and **Tier 3** (consistent-hash sharding) are
opt-in features built from source. Promotion of a standby and automatic failover are
planned. For connection routing at the wire layer, put the standalone
**HeliosProxy** in front of the nodes.

For a step-by-step setup, see the
[High Availability Hands-On Tutorial](https://www.heliosdb.com/docs/nano/guides/ha_hands_on_tutorial/) and
[High Availability Cluster with Docker Compose](https://www.heliosdb.com/docs/nano/guides/ha-cluster-tutorial/).

---

## Tier Matrix

| Tier | Cargo feature | Default? | Architecture | Conflict resolution | Best for |
|------|---------------|---------:|--------------|---------------------|----------|
| **Tier 1** | `ha-tier1` | yes | Warm standby (WAL streaming) | N/A (single primary) | Read replicas, standby copies |
| **Tier 2** | `ha-tier2` | no | Branch-based active-active | Vector clock + field-level merge | Multi-region writes, edge sync |
| **Tier 3** | `ha-tier3` | no | Consistent hash ring sharding | Per-shard | Horizontal scaling beyond one host |

### Optional add-ons

| Feature | Adds | Implies |
|---------|------|---------|
| `ha-dedup` | Content-addressed deduplication across nodes | — |
| `ha-ab-testing` | Branch-based experiment routing | — |
| `ha-branch-replication` | Selective branch sync to remote servers | `ha-tier2` |
| `ha-full` | Bundle of all of the above | `ha-tier1+2+3` + add-ons |

### Build recipes

```bash
# Default — Tier 1 only (warm standby + DR)
cargo build --release

# Add Tier 2 (multi-primary)
cargo build --release --features ha-tier2

# Full HA stack
cargo build --release --features ha-full
```

```toml
[dependencies]
heliosdb-nano = { version = "4", features = ["ha-tier2"] }
```

---

## Tier 1: Warm Standby

Active-passive replication: one primary accepts writes and streams its WAL to
standbys that serve reads.

### Architecture

```
+-------------+     WAL stream     +-------------+
|   Primary   | -----------------> |   Standby   |
|  (writes)   |   replication port |   (reads)   |
+-------------+                    +-------------+
```

### Availability

| Capability | Status |
|------------|--------|
| WAL streaming to one or more standbys | Available in the default build |
| Sync modes `async`, `semi-sync`, `sync` | Available |
| Standby reads over the PostgreSQL protocol | Available after catch-up |
| Standby catch-up after a restart | Available while the primary holds the missing WAL |
| Dataset history guard (a standby never attaches to another primary's data) | Available |
| Shared-token authentication (`--replication-token-file`) | Available |
| Writes on a standby | Refused (SQLSTATE 25006); forwarding to the primary is planned |
| Switchover and promotion | Planned; `SWITCHOVER TO` is refused |
| Automatic failover, observer quorum and split-brain fencing | Planned; `--replication-role observer` starts no services |
| Encrypted replication channel | Planned |
| Re-seeding a standby from a primary that already holds data | Planned |

The library contains failover, split-brain and switchover components
(`FailoverWatcher`, `SplitBrainProtector`, `SwitchoverCoordinator`), but the
`start` command does not use them, and they are not a supported promotion path.

### Sync Modes

`--sync-mode` on the primary decides whether a commit waits for a standby.

| Mode | Commit on the primary |
|------|-----------------------|
| `async` (default) | Returns once the primary has written it |
| `semi-sync` | Waits until at least one connected standby has received the entry, up to 10 seconds |
| `sync` | Same wait as `semi-sync` in the current release |

With no standby connected there is no wait. When the wait times out, the primary
logs a warning and the commit stands.

---

## Tier 2: Multi-Primary

Active-active replication with conflict resolution.

### Architecture

```
┌─────────────┐   Branch Sync   ┌─────────────┐
│  Region A   │ ←─────────────→ │  Region B   │
│  (Primary)  │                 │  (Primary)  │
└─────────────┘                 └─────────────┘
    ↓     ↓                       ↓     ↓
  Writes Reads                  Writes Reads
```

### Components

| Component | Description |
|-----------|-------------|
| `MultiPrimarySyncManager` | Coordinates multi-region sync |
| `ConflictMergeEngine` | Resolves write conflicts |
| `RegionCoordinator` | Manages region topology |

### Conflict Resolution Strategies

| Strategy | Description | Use Case |
|----------|-------------|----------|
| Last-Write-Wins | Timestamp-based | Simple, no conflicts visible |
| Branch-Wins | Prefer local changes | Low-latency local writes |
| Merge | Combine changes | Collaborative editing |
| Custom | User-defined logic | Complex business rules |

### Configuration

```rust
use heliosdb_nano::replication::{
    MultiPrimarySyncManager,
    ConflictResolution,
};

let sync = MultiPrimarySyncManager::new()
    .add_region("us-east", "us-east.example.com:5432")
    .add_region("eu-west", "eu-west.example.com:5432")
    .conflict_resolution(ConflictResolution::LastWriteWins)
    .build();
```

### Branch-Based Replication

Multi-primary uses HeliosDB Nano's branching for conflict-free merges:

```sql
-- Each region maintains its own branch
-- Sync merges branches across regions

-- Region A writes
INSERT INTO orders (id, total) VALUES (1, 100);

-- Region B writes (concurrent)
INSERT INTO orders (id, total) VALUES (2, 200);

-- After sync: both rows present in all regions
```

---

## Tier 3: Sharding

Horizontal scaling with consistent hashing.

### Architecture

```
                    ┌─────────────┐
                    │   Router    │
                    └──────┬──────┘
           ┌───────────────┼───────────────┐
           ↓               ↓               ↓
    ┌──────────┐    ┌──────────┐    ┌──────────┐
    │ Shard 1  │    │ Shard 2  │    │ Shard 3  │
    │  (0-33%) │    │ (34-66%) │    │ (67-100%)│
    └──────────┘    └──────────┘    └──────────┘
```

### Components

| Component | Description |
|-----------|-------------|
| `HashRing` | Consistent hashing for key distribution |
| `ShardRouter` | Routes queries to correct shard |
| `ReshardManager` | Online resharding with minimal downtime |
| `VectorPartitioner` | Special partitioning for vector data |

### Sharding Strategies

| Strategy | Description | Best For |
|----------|-------------|----------|
| Hash | Consistent hash of shard key | Even distribution |
| Range | Key ranges per shard | Time-series data |
| Geographic | Location-based routing | Multi-region |
| Vector | Centroid-based partitioning | Vector search |

### Configuration

```rust
use heliosdb_nano::replication::{HashRing, ShardRouter};

let ring = HashRing::new()
    .add_node("shard1.example.com:5432", 100)  // weight: 100
    .add_node("shard2.example.com:5432", 100)
    .add_node("shard3.example.com:5432", 100)
    .build();

let router = ShardRouter::new(ring)
    .shard_key("tenant_id")  // Shard by tenant
    .build();
```

### Vector Partitioning

Special support for vector workloads:

```rust
use heliosdb_nano::replication::{VectorPartitioner, CentroidManager};

let partitioner = VectorPartitioner::new()
    .dimensions(768)
    .num_centroids(16)  // 16 partitions based on vector similarity
    .build();

// Vectors routed to shard containing nearest centroid
```

### Resharding

Online resharding without downtime:

```rust
use heliosdb_nano::replication::ReshardManager;

let reshard = ReshardManager::new(ring)
    .target_shards(6)  // Scale from 3 to 6 shards
    .parallel_streams(4)
    .build();

reshard.execute().await?;  // Non-blocking migration
```

---

## Logical Replication

For selective table replication:

```rust
use heliosdb_nano::replication::{
    LogicalReplicationPipeline,
    TableFilter,
    ColumnMapping,
};

let pipeline = LogicalReplicationPipeline::new()
    .source("source.example.com:5432")
    .destination("dest.example.com:5432")
    .table_filter(TableFilter::include(&["users", "orders"]))
    .column_mapping(ColumnMapping::new()
        .rename("old_name", "new_name")
        .exclude("sensitive_column"))
    .build();

pipeline.start().await?;
```

---

## CLI Options

Start HeliosDB Nano in HA (Tier 1 warm standby) mode with the `start`
subcommand's replication flags:

```bash
# Token shared by all nodes: 32 random bytes, mode 0600, owned by the server user
head -c 32 /dev/urandom > /etc/heliosdb/replication.token
chmod 600 /etc/heliosdb/replication.token

# Primary: replication listener on the --listen address at --replication-port
heliosdb-nano start --data-dir /var/lib/heliosdb/primary --listen 10.0.0.10 \
    --auth scram-sha-256 --password-file /etc/heliosdb/password \
    --replication-role primary \
    --replication-port 5433 \
    --sync-mode semi-sync \
    --replication-token-file /etc/heliosdb/replication.token \
    --allow-insecure-replication-remote

# Standby: the data directory must not exist before the first start
heliosdb-nano start --data-dir /var/lib/heliosdb/standby --listen 10.0.0.11 \
    --auth scram-sha-256 --password-file /etc/heliosdb/password \
    --replication-role standby \
    --primary-host 10.0.0.10:5433 \
    --replication-token-file /etc/heliosdb/replication.token
```

The replication channel is not encrypted. A primary with a token on a non-loopback
address starts only with `--allow-insecure-replication-remote`; run that only on a
private network or through a protected tunnel (for example WireGuard).

| Flag | Default | Description |
|------|---------|-------------|
| `--replication-role` | `standalone` | `standalone`, `primary`, `standby` or `observer` |
| `--replication-port` | `5433` | Port used for WAL streaming |
| `--primary-host` | — | `host:port` of the primary's replication listener (standby) |
| `--sync-mode` | `async` | `async`, `semi-sync` or `sync` |
| `--replication-token-file` | — | Shared 32-byte token file (mode 0600) |
| `--allow-insecure-replication-remote` | off | Allow the token on a non-loopback listener (plaintext) |
| `--replication-startup-timeout-secs` | `30` | Time a standby has to verify the primary and catch up |
| `--node-id` | auto | Node UUID; generated if omitted |
| `--standby-hosts`, `--observer-hosts` | — | Accepted and validated; not used by the current release |

The same settings can go in the `[replication]` section of the configuration file
(`role`, `replication_port`, `primary_host`, `sync_mode`, `startup_timeout_secs`,
`token_file`, `node_id`).

Tier 2 (multi-primary) and Tier 3 (sharding) are configured through the
Rust API (see their Configuration sections above), not through `start` command-line flags.

---

## Docker Support

The official image `ghcr.io/heliosdatabase/heliosdb-nano` includes Tier 1. A
standby container needs one adjustment: the image entrypoint creates the data
directory before starting the server, and a standby refuses a data directory that
already exists, so standbys run the binary directly. The complete Compose file is in
[High Availability Cluster with Docker Compose](https://www.heliosdb.com/docs/nano/guides/ha-cluster-tutorial/).

---

## Writes on a Standby

A standby refuses write statements in every sync mode:

```
ERROR:  transparent forwarding is disabled for a history-guarded standby
HINT:  Connect directly to the primary for write operations.
```

The SQLSTATE is `25006`. Send writes to the primary. The check covers INSERT,
UPDATE, DELETE, CREATE, DROP, ALTER and TRUNCATE sent as simple queries; it is not a
complete read-only security boundary. Forwarding writes from a standby to the
primary (transparent write routing) is planned.

---

## Monitoring

### HA System Views

HeliosDB Nano provides SQL system views for monitoring HA configuration and replication metrics.

#### pg_replication_status

View node configuration and role:

```sql
SELECT * FROM pg_replication_status;
```

| Column | Description |
|--------|-------------|
| `node_id` | Unique identifier for this node |
| `role` | primary, standby, observer, or standalone |
| `sync_mode` | async, semi-sync, or sync |
| `is_read_only` | true on a standby |
| `current_lsn` | Current log sequence number |
| `standby_count` | Number of connected standbys (primary only) |
| `primary_host` | Configured primary endpoint (standby only) |
| `listen_addr` | Listen address |
| `replication_port` | Replication listener port |
| `started_at` | Time the node started |

#### pg_replication_standbys (Primary Only)

View connected standbys:

```sql
SELECT * FROM pg_replication_standbys;
```

| Column | Description |
|--------|-------------|
| `node_id` | Standby's unique identifier |
| `address` | Standby's connection address |
| `sync_mode` | Replication mode for this standby |
| `state` | connecting, streaming, catching_up, synced, disconnected |
| `current_lsn` | Standby's current LSN position |
| `flush_lsn` | Flushed LSN |
| `apply_lsn` | Applied LSN |
| `lag_bytes` | Replication lag in bytes |
| `lag_ms` | Replication lag in milliseconds |
| `connected_at` | Connection timestamp |
| `last_heartbeat` | Last heartbeat received |

#### pg_replication_primary (Standby Only)

View primary connection status:

```sql
SELECT * FROM pg_replication_primary;
```

| Column | Description |
|--------|-------------|
| `primary_node_id` | Primary's unique identifier |
| `primary_address` | Primary's address |
| `connection_state` | disconnected, connecting, connected, streaming, error |
| `primary_lsn` | Primary's current LSN |
| `local_lsn` | Local LSN position |
| `lag_bytes` | Replication lag in bytes |
| `lag_ms` | Replication lag in milliseconds |
| `fencing_token` | Token issued by the primary in the replication handshake |
| `connected_at` | Connection timestamp |
| `last_heartbeat` | Last heartbeat received |

#### pg_replication_metrics

View performance metrics:

```sql
SELECT * FROM pg_replication_metrics;
```

One row per metric:

| Column | Description |
|--------|-------------|
| `metric_name` | Name of the metric |
| `metric_value` | Current value |
| `description` | What the metric counts |

### Monitoring Examples

```sql
-- Check if standbys are in sync
SELECT
    node_id,
    CASE
        WHEN lag_ms < 1000 THEN 'IN_SYNC'
        WHEN lag_ms < 60000 THEN 'CATCHING_UP'
        ELSE 'LAGGING'
    END as status,
    lag_ms
FROM pg_replication_standbys;

-- View all nodes in cluster
SELECT node_id, role, current_lsn
FROM pg_replication_status;
```

---

## Best Practices

1. **Network**: Run replication over loopback, a private network or a protected tunnel; the channel is not encrypted.
2. **Token**: Always set `--replication-token-file` so only nodes with the token can attach as standbys.
3. **Monitoring**: Alert on `lag_bytes` / `lag_ms` in `pg_replication_standbys` and on standbys missing from it.
4. **Standby maintenance**: Bring a stopped standby back before the primary checkpoints past its position (by default 1,000 entries or 30 seconds of writes); otherwise it needs a rebuild.
5. **Backups**: Keep taking backups with `heliosdb-nano dump`; a standby is not a backup and cannot be promoted yet.

---

## HeliosProxy: Wire-Level Routing

For PostgreSQL-wire connection routing and read/write splitting between nodes,
deploy the standalone **HeliosProxy** in front of the nodes:

- **Repository:** [github.com/HeliosDatabase/HeliosDB-Proxy](https://github.com/HeliosDatabase/HeliosDB-Proxy)
- **Documentation:** [HeliosProxy](/docs/proxy/)
- **Topology:** sits between clients and the Nano nodes, speaks the PostgreSQL wire
  protocol, sends writes to the primary and reads to standbys.

Because Nano standbys cannot be promoted yet, a primary outage is an outage for
writes even behind the proxy.

---

## See Also

- [Configuration](https://www.heliosdb.com/docs/nano/user-guide/configuration/)
- [Deployment Modes](https://www.heliosdb.com/docs/nano/getting-started/deployment/)
- [Multi-Tenancy](https://www.heliosdb.com/docs/nano/features/multi-tenancy/)
- [BACKUP_RESTORE_TUTORIAL](https://www.heliosdb.com/docs/nano/guides/tutorials/backup_restore_tutorial/) — keep PITR running alongside HA
- [FIPS_COMPLIANCE_TUTORIAL](https://www.heliosdb.com/docs/nano/guides/tutorials/fips_compliance_tutorial/) — HA in regulated deployments