HeliosProxy 1.7.0 — Transaction Replay That Checks Its Own Work

A proxy that replays your transaction onto a new database after the old one dies is making a promise: what you already saw is still true. HeliosProxy 1.7.0 is the release where that promise stops being an assumption and becomes something the proxy verifies, statement by statement — and where it says so plainly when it cannot.

This closes a six-item correctness series that began with an adversarial audit of 1.6.0. Every defect that audit found in Transaction Replay is fixed and shipped.

It is core, not a feature flag

Worth correcting up front, because we have described this wrongly before: the recovery a live client experiences needs no cargo feature and no application change. Any PostgreSQL-wire client, any PG-wire backend. The ha-tr feature is something else — a write journal plus the operator-driven /api/replay and /api/shadow endpoints and the failover library components. Turning it off does not turn off in-session failover.

tr_mode selects how far recovery goes, and each level includes the one before it:

• none — nothing survives. One error, connection closed.
• session (default) — the connection and its session state. A statement that never reached the dead backend is re-run.
• select — plus: an interrupted read is re-executed, when that read is provably side-effect-free.
• transaction — plus: an uncommitted transaction is replayed from its BEGIN on the new primary and the interrupted statement re-run inside it.

Replay now verifies its own results

While a transaction is recorded, each statement's response frames are hashed in order into a bounded digest — the bytes as the client actually saw them. During replay the replacement backend's responses are hashed the same way. Any divergence rolls the replay back and returns 40001, instead of continuing a transaction on top of rows the client never observed.

Two refusals follow from the same principle. Transactions pinned to a SERIALIZABLE or REPEATABLE READ snapshot are never replayed, because no replacement backend can reproduce that snapshot. And a response larger than [limits] tr_max_observation_bytes has no digest to verify against, so its transaction is marked non-replayable rather than replayed on faith.

“It is a SELECT” is not a licence to run it twice

Re-executing an interrupted read used to be decided by the leading keyword. A SELECT can call nextval(), pg_notify(), set_config(), an advisory lock, or any user-defined function with arbitrary effects. Replay now scans every function call in the statement and re-executes only when each one is a PostgreSQL built-in known to be side-effect-free — or a name you added to the new tr_read_functions. Anything else is opaque: the client gets 08007 and the statement is never run a second time.

Session state, the way PostgreSQL actually behaves

A SET inside a transaction is not permanent until that transaction commits, and ROLLBACK TO SAVEPOINT undoes the ones made after the savepoint. Restore now models exactly that: operations are deferred to commit, savepoint-scoped on rollback, and keyed by variable name so repeating a SET reuses its slot instead of consuming another. A rolled-back RESET ALL no longer wipes the restore set. When tracked state exceeds its cap, the failover is refused with 08006 rather than silently re-homing a session with half its settings.

One deadline for the whole recovery

write_timeout_secs keeps its name and its 30-second default, but it is now a single deadline covering the entire recovery: waiting for a primary, connecting, authenticating, restoring session state, replaying. Previously each phase had its own timeout and they could add up well past the configured value.

Alongside it, every streaming relay now validates each backend frame header against [limits] max_backend_frame_bytes and closes the offending backend. A companion backend_response_timeout_secs bounds a response as a whole, which a per-read timeout can never do against a backend trickling bytes.

How it was tested

Correctness claims about failover are worth exactly as much as the failures you actually injected.

• Live commit-outcome suite, 7 cases. Real PostgreSQL 18.4 in a disposable container; the connection is destroyed while a COMMIT is in flight, then the suite asserts against the database's real state whether the client's report was correct. The pre-fix binary failed 4.
• Live streaming suite, 13 cases. Faults during large result sets, across Flush chunks, mid-DataRow, after CommandComplete. Asserts no client ever sees a duplicated row or a second protocol completion. The pre-fix binary failed 12.
• Synthetic boundary harness, 26 cases. Commit-boundary classification against comments, quoted and dollar-quoted text, multi-statement strings, backslash escapes, savepoint stacks.
• Deterministic partial-writer. A writer that fails after every byte offset, confirming no offset can produce a duplicate committed write.
• Adversarial review. Three independent reviewers, one per lens. That is how a relay publishing a chunk that ended mid-frame was caught.
• One recommendation was rejected. A reviewer proposed not closing the session after a published CommandComplete. It reads like an availability win, but injecting an error after a statement the backend actually finished tells the client a completed INSERT failed — and a client that retries it double-applies. The harness caught the regression when it was tried.
• Full gate matrix on the release commit. clippy -D warnings across four feature profiles; 432 / 518 / 1726 / 1727 tests on default, ha-tr, all-features and all-features,postgres-topology, zero failures; MSRV 1.86 with a locked dependency graph.

What it still refuses to do

Transparent recovery is the goal, a clear error is the fallback, and a wrong answer is never acceptable. The proxy declines, and says why, when:

• a COMMIT's outcome is genuinely unknown — never retried (08007, with guidance to verify);
• part of a response already reached the client — the session closes rather than concatenate a second result;
• the transaction ran at SERIALIZABLE or REPEATABLE READ;
• a response was too large to have been digested within tr_max_observation_bytes;
• an interrupted read calls anything outside the side-effect-free set;
• tracked session state exceeded its cap (08006).

Known limits, stated plainly: prepared statements, session temp tables and open cursors are not restored across a failover, and SETs issued through the extended protocol — as most JDBC drivers send them — are not tracked. Replay does not make now(), random() or a RETURNING serial reproduce their first values; verification catches divergent results, it does not create determinism. The ha-tr journal behind /api/replay is in-memory and best-effort; hardening it into a durable, ordered, recovery-grade log is the next item in this series.

New configuration

tr_read_functions = []               # extra provably-pure functions
write_timeout_secs = 30              # now ONE deadline for the whole recovery

[limits]
tr_max_observation_bytes = 1048576   # replay-verification digest budget
max_backend_frame_bytes = 104857600  # ceiling on one backend frame
backend_response_timeout_secs = 0    # 0 = off; bounds a slow-dripping response

Upgrading

Drop-in from 1.6.x — no configuration changes are required and the new limits have safe defaults. Sessions using tr_mode = "transaction" will see replay refused in cases where it previously proceeded: snapshot-pinned transactions, unverifiable responses. That is the point of the release, and each refusal carries a specific SQLSTATE.

Source, changelog and the full Transaction Replay reference: github.com/dimensigon/HDB-HeliosDB-Proxy.