Production Deployment: Backup & Disaster Recovery

Part of: Production Deployment Guide

HeliosDB Full keeps all of its state in one data directory (--data-dir, environment HELIOSDB_STORAGE_DATA_DIR; /var/lib/heliosdb/data in the container image): the write-ahead log under wal/, the SSTables, and the catalog. A backup is therefore a consistent copy of that directory, plus the server’s flags and environment file so the copy can be started the same way. This page describes the methods that work with the current server and how to restore from them.

8.1 Backup Strategies

Durability you get without a backup

With the default --synchronous-commit on, each acknowledged write is fsynced to the WAL before the client sees success. After a crash, power loss or kill -9, the server replays the WAL on start and every committed transaction is present. With --synchronous-commit off, the loss window after a crash is bounded by --wal-flush-interval-ms (default 5 ms). Neither protects against losing the disk or deleting data by mistake; that is what backups are for.

Method 1: Cold copy (simplest, fully consistent)

Stop the server, copy the directory, start it again. On SIGTERM the server flushes its memtable and logs Storage flushed successfully — all data persisted before it exits.

systemctl stop heliosdb                    # or: docker stop --time 60 heliosdb
tar -C /var/lib -czf /backup/heliosdb-$(date +%Y%m%d-%H%M).tar.gz heliosdb
systemctl start heliosdb

Use this when a short maintenance window is acceptable.

Method 2: Storage snapshot (online, crash-consistent)

An atomic snapshot of the volume that holds the data directory captures the same state a power loss would leave, which the WAL replay recovers. Take it with the storage layer you already run:

# LVM
lvcreate --snapshot --size 20G --name heliosdb-snap /dev/vg0/heliosdb
mount -o ro /dev/vg0/heliosdb-snap /mnt/heliosdb-snap
tar -C /mnt/heliosdb-snap -czf /backup/heliosdb-$(date +%Y%m%d-%H%M).tar.gz .
umount /mnt/heliosdb-snap && lvremove -y /dev/vg0/heliosdb-snap

# ZFS
zfs snapshot tank/heliosdb@$(date +%Y%m%d-%H%M)

# Cloud block storage: an EBS / Persistent Disk / Managed Disk snapshot of the data volume
# Kubernetes: a VolumeSnapshot of the data PersistentVolumeClaim

The snapshot must cover the whole data directory, including wal/, in one atomic operation. A file-by-file copy of a running server (cp, rsync without a snapshot) is not consistent; use Method 1 or 2.

Method 3: Logical export (selected tables, portable)

Export table data with any client and keep the DDL under version control:

# Data, one CSV file per table
psql "host=db.internal user=postgres" --csv -c "SELECT * FROM orders" > orders.csv

# Re-import into HeliosDB (COPY ... FROM STDIN is supported)
psql "host=db.internal user=postgres" -c "\copy orders FROM 'orders.csv' WITH (FORMAT csv, HEADER true)"

COPY ... TO STDOUT (and therefore \copy ... TO) is not available in the current release; use SELECT with --csv as above, or your ETL tool.

Schedule and retention

A common pattern: a nightly snapshot or cold copy kept for 7 to 30 days, a weekly copy moved to another region or account, and a logical export of the most important tables before each schema change. Encrypt backups at rest in the backup target (object-store server-side encryption, an encrypted backup volume, or gpg/age on the archive) and store the environment file with the server’s secrets separately from the data.

# Example: nightly cold copy to object storage (cron, 02:30)
30 2 * * * systemctl stop heliosdb && \
  tar -C /var/lib -czf - heliosdb | aws s3 cp - s3://example-heliosdb-backups/$(hostname)/$(date +\%F).tar.gz && \
  systemctl start heliosdb

8.2 Restore and Point-in-Time Recovery

Restore a copy

systemctl stop heliosdb
mv /var/lib/heliosdb /var/lib/heliosdb.old
mkdir -p /var/lib/heliosdb
tar -C /var/lib -xzf /backup/heliosdb-20261001-0230.tar.gz
chown -R heliosdb:heliosdb /var/lib/heliosdb
systemctl start heliosdb
psql "host=127.0.0.1 user=postgres" -c "SELECT COUNT(*) FROM orders"

The restored server comes up with the data as of the copy. If you restore onto another host, start it with the same flags and the same HELIOSDB_SUPERUSER_PASSWORD and listener credentials.

Point in time

The recovery points are the copies you keep. The current server has no WAL archiving or AS OF time travel, so restoring to an arbitrary moment between two backups is not available; shorten the backup interval (snapshots are cheap) if you need a smaller recovery point objective. Statements such as SHOW BACKUP STATUS, CREATE SNAPSHOT or ALTER SYSTEM SET wal_archiving return SQLSTATE 0A000.

Test restores

Restore a recent backup into a scratch directory on a regular schedule and check it:

mkdir /tmp/restore-test && tar -C /tmp/restore-test -xzf /backup/latest.tar.gz
HELIOSDB_SUPERUSER_PASSWORD="$(cat /run/secrets/heliosdb-superuser)" \
  heliosdb-full --data-dir /tmp/restore-test/heliosdb --postgres-port 55432 --no-http &
psql "host=127.0.0.1 port=55432 user=postgres" -c "SELECT COUNT(*) FROM orders"
kill %1

8.3 Disaster Recovery Across Sites

The heliosdb-full server runs as a single node: it does not stream replication or fail over to another node by itself (SHOW REPLICATION STATUS and FAILOVER return 0A000). Build site-level recovery from the backup methods above:

Warm standby by restore. Ship snapshots or cold copies to the second site on a schedule. To fail over, restore the latest copy there and start the server; point clients at it through DNS or your load balancer.
Replicated storage. Run the data directory on storage that replicates synchronously between zones (for example a regional persistent disk). After a zone failure, start the server in the surviving zone on the same volume. Only one server may use a data directory at a time.
Application-level dual writes or change feeds where the application already has them.

Document the recovery steps, the recovery point (backup interval) and the expected recovery time, and rehearse them.