HeliosProxy 2.0.0 - Committed History, Patroni Failover, and a Cache That Waits for COMMIT

Two releases since the last post. HeliosProxy 1.9 (September 23) gave the proxy a record of what the backend actually committed. 2.0 (September 25) makes the result cache, the failover topology and Transaction Replay all work from that record — and it is a major version because a few defaults now behave differently.

1.9: a journal of committed transactions

The old write journal stored SQL text after the response had gone out, as synthetic transactions nobody checked against the backend. 1.9 replaces it with a per-session capture that reconciles what was sent with what the backend answered, on both wire protocols: real BEGIN…COMMIT boundaries as the backend reported them, per-statement outcomes, savepoint structure, bound parameters byte for byte in their original text or binary format, the source (client, user, database, backend, tenant) and a global commit sequence. Rolled-back work, rejected statements and two-phase transactions never enter committed history.

With [journal] dir set, committed transactions are appended to a segmented on-disk journal — CRC-checked records, torn-tail truncation, rotation, size-based retention — and reloaded at startup. The data path never waits on the disk: records go through a bounded writer queue under a commit, interval or none fsync policy.

[journal]
dir = "/var/lib/heliosproxy/journal"   # empty = memory only
max_committed_transactions = 50000     # kept in memory, reloaded at startup
fsync = "interval"                     # "commit" | "interval" | "none"
fsync_interval_ms = 100

POST /api/replay gains "mode": "committed_history": committed transactions in commit order, one BEGIN…COMMIT per transaction on one connection, parameters re-sent in their captured format, stopping at the first failure with a last_commit_seq to resume from.

Measured where users feel it

The first version of that journal passed our micro-benchmark gate at +2.97%. Driven with pgbench through the proxy against PostgreSQL 18.4 at 16 clients, it cost 40% of committed write throughput: every explicit transaction took a shared asynchronous lock four times, and under concurrency each acquisition parked the task. Journaling each transaction session-locally and touching shared state once, at commit, took the write path to 15.2k TPS — above the 13.3k it did before the journal existed.

That user-path run — committed TPS and p99 latency at 16 and 64 clients, three interleaved passes, a 3% budget per write cell — is now a mandatory gate for write-path changes and for every release. It needed one fix of its own: PostgreSQL's timed checkpoints, when they landed inside a measured cell, froze every client for 100–300 ms at a time and read as a 16% regression in whichever binary they hit. The harness now takes a checkpoint and warms up before each write cell, and records how many checkpoints started during it.

2.0: the cache waits for COMMIT

Before 2.0 the query cache was invalidated only by writes sent through the simple query protocol. Prepared statements — what most drivers and ORMs send — did not invalidate it at all, and the per-connection and semantic tiers were bounded only by their TTL.

2.0 drives invalidation from the same capture that feeds the journal, so it sees what the backend executed on both protocols:

• Every cached result records the write generation of the tables it read, taken before its backend fetch, and every hit on every tier is checked against it.
• An autocommit write moves its tables' generation before the client sees the ReadyForQuery that acknowledges it. Inside an explicit transaction the tables are staged and move when the backend reports the commit; a rollback drops them.
• A result whose tables were written while it was being fetched is not stored, which closes the reader-refill race.
• DDL, TRUNCATE, COPY … FROM, EXECUTE, CALL and DO invalidate everything.

Correctness was the point, but throughput moved too. The old invalidation walked a written table's cached keys on every commit: after a read-heavy phase the first writes stalled for seconds, and turning the cache on cost up to a third of write throughput. On the user-path gate with the cache on, committed-write throughput is up 23–45% against 1.9 and roughly level with running without a cache; p99 latency is flat or lower.

Why the cache is still opt-in

A cached result is identified by its SQL, database, user and branch — not by session state. Two sessions of the same user that differ only in SET ROLE, search_path or a setting read by row-level-security policies (a tenant id set per request on a shared connection user, for example) could be served each other's result. Writes that bypass this proxy, and COMMIT PREPARED, are still bounded only by ttl_secs. The proxy's own multi-tenancy filtering is safe, because it rewrites the SQL per tenant before the lookup. So [cache] enabled stays false by default until session state is part of the cache identity.

Patroni as the write authority

[topology]
provider = "patroni"
patroni_endpoints = ["http://pg-a:8008", "http://pg-b:8008", "http://pg-c:8008"]
patroni_request_timeout_ms = 2000
lease_timeout_secs = 10

The proxy polls GET /cluster and routes writes to the running leader as Patroni's DCS sees it, so a promotion moves writes without editing proxy.toml. Only a running leader counts, a standby_leader never does, and the leader must be one of your configured nodes. Two running leaders authorize nothing, and when no endpoint answers, writes stop at the lease boundary instead of continuing on stale knowledge.

The built-in PostgreSQL provider follows the same fail-closed rule: when several nodes claim to be writable, the one on the strictly highest timeline wins and a tie authorizes nothing. GET /topology reports the leader's timeline and a conflictingPrimariesTotal count, also exported to Prometheus.

How it was tested

• A two-client stale-read test against live PostgreSQL: an autocommit prepared-statement write, a transaction another client refills around before it commits, a rollback, a multi-statement string, DDL and COPY — no stale value served in any of them.
• Patroni end to end against a mock /cluster endpoint: switchover, split brain, a dead first endpoint. That test caught a real bug: the primary tracker followed a new leader only through a change event, so a missed event left it on the old primary until its lease ran out. It now reconciles on every check.
• The user-path gate with the cache off (within 0.2% of 1.9) and on (above).
• The full gate matrix on the release tree: clippy -D warnings on five feature profiles; 593 / 677 / 677 / 1923 / 1926 tests, zero failures; 50/50 live integration tests; MSRV 1.86 with a locked dependency graph.

Upgrading

Most deployments drop in. The major version is for behaviour changes worth reading first:

• The HTTP, GraphQL and MCP gateways now admit at most max_concurrent_requests = 64 requests at once and answer 503 with Retry-After beyond that; query_timeout_ms replaces the fixed 30-second timeout.
• Unknown configuration keys are reported at any nesting depth; strict_config = true turns them into a startup error.
• Conflicting primaries fail closed, as above.
• With the cache enabled, stale entries are refused on every tier, and writes whose tables cannot be determined clear it.

1.9 also moved TLS to rustls 0.23.45 (RUSTSEC-2026-0285) and replaced two unmaintained dependencies; the advisory count is zero.

cargo install heliosdb-proxy --version 2.0.0
docker pull ghcr.io/heliosdatabase/hdb-heliosdb-proxy:2.0.0

Source, changelog and upgrade notes: github.com/HeliosDatabase/HeliosDB-Proxy.