Skip to content
HeliosDB HeliosDB Full

Production Deployment: Troubleshooting

Part of: Production Deployment Guide

Every diagnostic on this page uses the server log, psql (or any PostgreSQL client), curl against the HTTP gateway, and the server’s own flags. See the Server Configuration Reference for every flag mentioned here.

9.1 Common Issues

The server does not start

Read the last lines of the log (journalctl -u heliosdb -n 50, docker logs heliosdb). The server states the reason and the fix:

Log messageCauseFix
Refusing to bind '0.0.0.0' (non-loopback) without explicit opt-in--bind-addr is not a loopback addressAdd --allow-external (or HELIOSDB_ALLOW_EXTERNAL=true) after configuring authentication and a firewall
Refusing to start with --auth-mode trust while binding ...trust on an external addressUse --auth-mode scram, or bind to 127.0.0.1
Configuration sets options heliosdb-full does not honourThe TOML file has keys the server does not readRemove them (the message lists them and the honoured keys) or set HELIOSDB_CONFIG_ALLOW_UNSUPPORTED=1
HELIOSDB_DATA_DIR=... is not read by heliosdb-fullLegacy variable from an old environment fileUse HELIOSDB_STORAGE_DATA_DIR (or --data-dir) and unset HELIOSDB_DATA_DIR
PostgreSQL TLS: cannot load ...Certificate or key unreadable or invalidCheck the paths, file permissions and PEM format
Failed to bind PostgreSQL port ...: Address already in useAnother process owns the portss -ltnp 'sport = :5432'; change the port flag or stop the other process
No protocols enabled!Every listener disabled or failed to bindCheck the earlier Failed to bind lines

Clients cannot connect

  1. Check the listener is open. The startup log prints one line per listener, for example MySQL protocol listening on 10.0.0.12:3306. MySQL, Oracle, SQL Server, MongoDB, Cassandra and Redis are off until enabled with --enable-<protocol>.
  2. Check the address. The default bind is 127.0.0.1; remote clients need --bind-addr with --allow-external, and the container port must be published.
  3. Check authentication:
    • PostgreSQL: user postgres with HELIOSDB_SUPERUSER_PASSWORD, or a role created with CREATE ROLE ... LOGIN PASSWORD. If the variable was not set, the generated password is printed once in the first startup log.
    • A VALID UNTIL in the past or a CONNECTION LIMIT that is reached refuses the login; SHOW ROLES and SHOW CONNECTIONS show both.
    • Other listeners use their own variables; see Authentication and credentials.
  4. With --tls-require (native PostgreSQL TLS, planned for the next release; current listeners do not terminate TLS), clients must connect with sslmode=require or stricter.
  5. Rate limits: more than 500 new connections per minute from one IP are refused by default (--conn-rate-limit), and the total is capped by --max-connections (default 10000). Connection pools in front of the database avoid both.

High query latency

-- What is running right now
SHOW CONNECTIONS;
SELECT pid, usename, state, query FROM pg_stat_activity;


-- Plan of a slow statement
EXPLAIN ANALYZE SELECT ... ;

Statements slower than --log-min-duration-statement (default 1000 ms) are logged at WARN with their duration; lower the threshold for a session with SET log_min_duration_statement = 200; while you investigate.

Common fixes:

  1. Indexes. CREATE INDEX builds and maintains an index only when the server runs with --enable-secondary-indexes; without it, CREATE INDEX records catalog metadata only. Enable the flag, then create the indexes your WHERE and join columns need.
  2. Read cache. --read-cache-mb (default 0, off) caches decoded SSTable blocks for repeated reads and scans; start with a quarter of the memory you can spare.
  3. Compaction. Without --enable-compaction, SSTables accumulate and reads touch more files over time. Enable it on long-running servers; cap its I/O with --compaction-max-mbps if foreground latency matters more.
  4. Write latency. Each commit is fsynced by default. Batch inserts in transactions or multi-row INSERTs; use --synchronous-commit off only where losing the last few milliseconds of writes after a crash is acceptable.

Out of memory

The memtable (64 MB by default, memtable_size_mb in the config file or HELIOSDB_STORAGE_MEMTABLE_SIZE_MB), the read cache (--read-cache-mb) and per-query working memory make up most of the server’s memory. To reduce it:

  1. Lower --read-cache-mb.
  2. Lower --max-connections and put a connection pool in front of the server.
  3. Set statement_timeout for ad-hoc or reporting users so runaway queries end.
  4. Give the container or systemd unit a memory limit with headroom above these sizes, so the kernel does not kill the process under pressure.

Disk usage grows

  1. Enable background compaction (--enable-compaction) so overwritten and deleted rows are reclaimed.
  2. Check SHOW STORAGE STATUS and the size of <data-dir>/wal.
  3. The periodic memtable flush (--flush-interval-secs, default 60) keeps the WAL bounded; do not set it to 0 on long-running servers.

A statement returns SQLSTATE 0A000

The statement belongs to a subsystem that this server does not run, for example SHOW REPLICATION STATUS, FAILOVER, GRANT, CREATE SNAPSHOT or ALTER TABLE ... ENABLE ROW LEVEL SECURITY. The message names the subsystem and nothing was changed. heliosdb-full --show-features --all lists them.

9.2 Debug Procedures

More logging

Terminal window
# Debug logging for the whole server
heliosdb-full --verbose ...


RUST_BACKTRACE=1                      # backtraces on errors (set in the container image)
-- Log every statement of this session with its duration
SET log_min_duration_statement = 0;

Collect a diagnostic bundle

Terminal window
mkdir -p /tmp/heliosdb-diag && cd /tmp/heliosdb-diag
heliosdb-full --version                  > version.txt
heliosdb-full --show-features --all      > features.txt
journalctl -u heliosdb --since "1 hour ago" > server.log     # or: docker logs --since 1h heliosdb
curl -s http://127.0.0.1:8443/health     > health.json
psql "host=127.0.0.1 user=postgres" -c "SHOW SERVER STATUS" -c "SHOW PROTOCOLS" \
  -c "SHOW CONNECTIONS" -c "SHOW STORAGE STATUS" -c "SHOW WAL STATUS" > status.txt
du -sh /var/lib/heliosdb /var/lib/heliosdb/wal > disk.txt
tar -czf ../heliosdb-diag-$(date +%Y%m%d-%H%M).tar.gz .

Remove secrets before sharing logs: the environment file is not part of the bundle, and the server never logs the superuser password when you set it through HELIOSDB_SUPERUSER_PASSWORD.

Health checks

Terminal window
curl -fsS http://127.0.0.1:8443/health    # {"status":"healthy","version":"..."}
curl -fsS http://127.0.0.1:8443/ready     # {"ready":true}
pg_isready -h 127.0.0.1 -p 5432

9.3 Performance Tuning

GoalSetting
Faster repeated reads--read-cache-mb 1024 (with --prefetch-depth for scans)
Stable read latency over time--enable-compaction, --compaction-max-mbps 200
Index lookups--enable-secondary-indexes, then CREATE INDEX
Higher write throughputBatch writes; --synchronous-commit off if the bounded loss window is acceptable
Protect against runaway queriesSET statement_timeout at session start (most connection pools have an init-SQL hook)
Protect against connection storms--conn-rate-limit, --max-connections, a pooler

The Performance Tuning Guide covers each setting in more detail.