---
title: "Production Deployment: Troubleshooting"
---

# Production Deployment: Troubleshooting

**Part of**: [Production Deployment Guide](https://www.heliosdb.com/docs/full/deployment/production_deployment_guide/)

Every diagnostic on this page uses the server log, `psql` (or any PostgreSQL client),
`curl` against the HTTP gateway, and the server's own flags. See the
[Server Configuration Reference](/docs/full/reference/configuration/) for every flag
mentioned here.

---

## 9.1 Common Issues

### The server does not start

Read the last lines of the log (`journalctl -u heliosdb -n 50`, `docker logs heliosdb`).
The server states the reason and the fix:

| Log message | Cause | Fix |
|-------------|-------|-----|
| `Refusing to bind '0.0.0.0' (non-loopback) without explicit opt-in` | `--bind-addr` is not a loopback address | Add `--allow-external` (or `HELIOSDB_ALLOW_EXTERNAL=true`) after configuring authentication and a firewall |
| `Refusing to start with --auth-mode trust while binding ...` | `trust` on an external address | Use `--auth-mode scram`, or bind to `127.0.0.1` |
| `Configuration sets options heliosdb-full does not honour` | The TOML file has keys the server does not read | Remove them (the message lists them and the honoured keys) or set `HELIOSDB_CONFIG_ALLOW_UNSUPPORTED=1` |
| `HELIOSDB_DATA_DIR=... is not read by heliosdb-full` | Legacy variable from an old environment file | Use `HELIOSDB_STORAGE_DATA_DIR` (or `--data-dir`) and unset `HELIOSDB_DATA_DIR` |
| `PostgreSQL TLS: cannot load ...` | Certificate or key unreadable or invalid | Check the paths, file permissions and PEM format |
| `Failed to bind PostgreSQL port ...: Address already in use` | Another process owns the port | `ss -ltnp 'sport = :5432'`; change the port flag or stop the other process |
| `No protocols enabled!` | Every listener disabled or failed to bind | Check the earlier `Failed to bind` lines |

### Clients cannot connect

1. Check the listener is open. The startup log prints one line per listener, for
   example `MySQL protocol listening on 10.0.0.12:3306`. MySQL, Oracle, SQL Server,
   MongoDB, Cassandra and Redis are off until enabled with `--enable-<protocol>`.
2. Check the address. The default bind is `127.0.0.1`; remote clients need
   `--bind-addr` with `--allow-external`, and the container port must be published.
3. Check authentication:
   - PostgreSQL: user `postgres` with `HELIOSDB_SUPERUSER_PASSWORD`, or a role created
     with `CREATE ROLE ... LOGIN PASSWORD`. If the variable was not set, the generated
     password is printed once in the first startup log.
   - A `VALID UNTIL` in the past or a `CONNECTION LIMIT` that is reached refuses the
     login; `SHOW ROLES` and `SHOW CONNECTIONS` show both.
   - Other listeners use their own variables; see
     [Authentication and credentials](/docs/full/reference/configuration/#authentication-and-credentials).
4. With `--tls-require` (native PostgreSQL TLS, planned for the next release; current listeners do not terminate TLS), clients must connect with `sslmode=require` or stricter.
5. Rate limits: more than 500 new connections per minute from one IP are refused by
   default (`--conn-rate-limit`), and the total is capped by `--max-connections`
   (default 10000). Connection pools in front of the database avoid both.

### High query latency

```sql
-- What is running right now
SHOW CONNECTIONS;
SELECT pid, usename, state, query FROM pg_stat_activity;

-- Plan of a slow statement
EXPLAIN ANALYZE SELECT ... ;
```

Statements slower than `--log-min-duration-statement` (default 1000 ms) are logged at
WARN with their duration; lower the threshold for a session with
`SET log_min_duration_statement = 200;` while you investigate.

Common fixes:

1. **Indexes.** `CREATE INDEX` builds and maintains an index only when the server
   runs with `--enable-secondary-indexes`; without it, `CREATE INDEX` records catalog
   metadata only. Enable the flag, then create the indexes your `WHERE` and join
   columns need.
2. **Read cache.** `--read-cache-mb` (default 0, off) caches decoded SSTable blocks
   for repeated reads and scans; start with a quarter of the memory you can spare.
3. **Compaction.** Without `--enable-compaction`, SSTables accumulate and reads touch
   more files over time. Enable it on long-running servers; cap its I/O with
   `--compaction-max-mbps` if foreground latency matters more.
4. **Write latency.** Each commit is fsynced by default. Batch inserts in transactions
   or multi-row `INSERT`s; use `--synchronous-commit off` only where losing the last
   few milliseconds of writes after a crash is acceptable.

### Out of memory

The memtable (64 MB by default, `memtable_size_mb` in the config file or
`HELIOSDB_STORAGE_MEMTABLE_SIZE_MB`), the read cache (`--read-cache-mb`) and per-query
working memory make up most of the server's memory. To reduce it:

1. Lower `--read-cache-mb`.
2. Lower `--max-connections` and put a connection pool in front of the server.
3. Set `statement_timeout` for ad-hoc or reporting users so runaway queries end.
4. Give the container or systemd unit a memory limit with headroom above these sizes,
   so the kernel does not kill the process under pressure.

### Disk usage grows

1. Enable background compaction (`--enable-compaction`) so overwritten and deleted
   rows are reclaimed.
2. Check `SHOW STORAGE STATUS` and the size of `<data-dir>/wal`.
3. The periodic memtable flush (`--flush-interval-secs`, default 60) keeps the WAL
   bounded; do not set it to `0` on long-running servers.

### A statement returns SQLSTATE 0A000

The statement belongs to a subsystem that this server does not run, for example
`SHOW REPLICATION STATUS`, `FAILOVER`, `GRANT`, `CREATE SNAPSHOT` or
`ALTER TABLE ... ENABLE ROW LEVEL SECURITY`. The message names the subsystem and
nothing was changed. `heliosdb-full --show-features --all` lists them.

---

## 9.2 Debug Procedures

### More logging

```bash
# Debug logging for the whole server
heliosdb-full --verbose ...

RUST_BACKTRACE=1                      # backtraces on errors (set in the container image)
```

```sql
-- Log every statement of this session with its duration
SET log_min_duration_statement = 0;
```

### Collect a diagnostic bundle

```bash
mkdir -p /tmp/heliosdb-diag && cd /tmp/heliosdb-diag
heliosdb-full --version                  > version.txt
heliosdb-full --show-features --all      > features.txt
journalctl -u heliosdb --since "1 hour ago" > server.log     # or: docker logs --since 1h heliosdb
curl -s http://127.0.0.1:8443/health     > health.json
psql "host=127.0.0.1 user=postgres" -c "SHOW SERVER STATUS" -c "SHOW PROTOCOLS" \
  -c "SHOW CONNECTIONS" -c "SHOW STORAGE STATUS" -c "SHOW WAL STATUS" > status.txt
du -sh /var/lib/heliosdb /var/lib/heliosdb/wal > disk.txt
tar -czf ../heliosdb-diag-$(date +%Y%m%d-%H%M).tar.gz .
```

Remove secrets before sharing logs: the environment file is not part of the bundle,
and the server never logs the superuser password when you set it through
`HELIOSDB_SUPERUSER_PASSWORD`.

### Health checks

```bash
curl -fsS http://127.0.0.1:8443/health    # {"status":"healthy","version":"..."}
curl -fsS http://127.0.0.1:8443/ready     # {"ready":true}
pg_isready -h 127.0.0.1 -p 5432
```

---

## 9.3 Performance Tuning

| Goal | Setting |
|------|---------|
| Faster repeated reads | `--read-cache-mb 1024` (with `--prefetch-depth` for scans) |
| Stable read latency over time | `--enable-compaction`, `--compaction-max-mbps 200` |
| Index lookups | `--enable-secondary-indexes`, then `CREATE INDEX` |
| Higher write throughput | Batch writes; `--synchronous-commit off` if the bounded loss window is acceptable |
| Protect against runaway queries | `SET statement_timeout` at session start (most connection pools have an init-SQL hook) |
| Protect against connection storms | `--conn-rate-limit`, `--max-connections`, a pooler |

The [Performance Tuning Guide](/docs/full/guides/user/performance_tuning_guide/)
covers each setting in more detail.

---

## Navigation

- **Previous**: [Backup & Disaster Recovery](https://www.heliosdb.com/docs/full/deployment/production_deployment_backup/)
- **Next**: [Advanced Deployment Scenarios](https://www.heliosdb.com/docs/full/deployment/production_deployment_advanced/)
- **Index**: [Production Deployment Guide](https://www.heliosdb.com/docs/full/deployment/production_deployment_guide/)