Skip to content
HeliosDB HeliosProxy

Transaction Replay (TR) — In-Session Failover

Transaction Replay (TR) — In-Session Failover

Transaction Replay keeps client sessions alive through a primary failure: sessions re-home and restore state, an interrupted read re-runs when it is provably side-effect-free, and opt-in transaction mode replays uncommitted work on the new primary — verifying every replayed response against what the client already saw, and never retrying a COMMIT whose outcome is unknown.

In-session replay is core. It needs no cargo feature and no application change — any PostgreSQL-wire client, any PG-wire backend. The ha-tr feature is a separate thing: the write journal plus the operator-driven /api/replay and /api/shadow endpoints and the failover library components. A build without ha-tr still does everything on this page.

As of HeliosProxy 1.7.0, replay verifies its own results. This page states exactly what each mode does, what is always true regardless of mode, what the proxy refuses to do, and the requirements and limits you must plan for. It does not claim zero data loss for in-flight COMMITs — see What is always true — and it does not claim cursor, temp-table or PREPARE survival; see What is never restored.

Why it matters

In a plain PostgreSQL HA setup, when the primary fails the pool drops active connections, in-flight statements get a connection-reset error, and every application must reconnect and decide on its own whether to retry. Most do not do this correctly.

With Transaction Replay the proxy is the reconnect-and-retry layer: the client connection stays open, the proxy re-homes it to the new primary, restores the session state, and — depending on tr_mode — re-executes what can safely be re-executed. Anything whose outcome cannot be established is reported to the client with a precise SQLSTATE instead of being guessed.

In-session replay (the core path)

This is what a live client experiences when its backend dies mid-session. It runs in core, in builds with and without ha-tr, and is independent of the ha-tr journal and the administrative replay endpoints.

What is recorded. While a session is inside a transaction under tr_mode = select or transaction, the proxy records each statement, the session state it changed, and a bounded digest of the response frames the client was shown ([limits] tr_max_observation_bytes, default 1 MiB). Autocommit traffic records nothing.

What happens on a backend fault. The proxy classifies how far the statement got. If the outcome is not delivered, recovery is safe. If the outcome is unknown — the statement may have committed — it is never re-executed. Once any part of a response has reached the client, the session is closed rather than have a second result appended to the first.

What recovery does. Under one deadline (write_timeout_secs) it waits for a healthy primary, connects and authenticates, restores the session’s tracked SET state, and replays the recorded transaction. Each replayed statement’s response is hashed and compared with what the client originally saw; any divergence rolls the replay back and returns 40001.

Behaviour by mode (tr_mode)

tr_modeWhat the client sees when the primary fails
noneA proper PostgreSQL error (SQLSTATE 57P01, admin shutdown) and the connection closes. Equivalent to a plain pooler.
session (default)The client connection stays alive. The proxy re-homes it to the new primary and restores SET/GUC state. The interrupted statement returns one error: 57P01 if it never reached the old primary, 08007 (transaction resolution unknown) if its outcome is unknown. Autocommit statements that had not yet been delivered are re-executed transparently.
selectEverything in session, plus an interrupted read is re-run on the new primary — but only when every function it calls is provably side-effect-free (a PostgreSQL built-in on the proxy’s allowlist, or a name you listed in tr_read_functions). A read calling a user-defined function, nextval, pg_notify, set_config, an advisory lock, a quoted-identifier call, or using SELECT … INTO is opaque: it returns 08007 and is never run twice. Read-only explicit transactions are replayed from BEGIN.
transaction (opt-in)Everything in select, plus the uncommitted transaction is replayed from BEGIN on the new primary and the in-flight statement re-executed inside it. Replay is verified: each statement’s original response frames were digested as the client saw them (within [limits] tr_max_observation_bytes), the replacement backend’s responses are digested the same way, and any divergence rolls the replay back and returns 40001 rather than continuing on top of rows the client never observed. A failed replayed statement produces the same 40001.

What is always true

These hold in every mode, including transaction:

  • A COMMIT whose outcome is unknown is never retried. The client receives 08007 and must check before retrying. This is what prevents double-apply; it also means Transaction Replay does not promise zero data loss for a COMMIT that was in flight at the moment of failure.
  • One deadline covers the whole recovery. Since 1.7.0 write_timeout_secs bounds waiting for a healthy primary, connect and authentication, session-state restore, replay and re-execution together — previously each phase carried its own timeout and they could add up well past the configured value. If no primary appears in the window the client gets 08006.
  • One error, then a clean transaction state. When the single error is returned inside an explicit transaction, the client-visible transaction is aborted (25P02 until the client issues ROLLBACK), exactly as PostgreSQL would.
  • A replay that fails yields 40001 (serialization failure), so standard retry logic applies; the session is not left half-replayed. The same 40001 is returned when replay verification detects divergence.
  • Protocol-level named prepared statements are re-prepared lazily on the new primary as they are next used. This is the extended-protocol Parse-with-a-name case only — SQL-level PREPARE foo AS … is not restored. See What is never restored.
  • A COPY in progress always fails with 08006 (connection failure). COPY streams are not journaled or resumed.
  • The proxy returns real PostgreSQL SQLSTATEs (57P01, 08007, 08006, 40001) so existing driver and application retry logic keeps working.

What it refuses to do

Transparent recovery is the goal, a clear error is the fallback, and a wrong answer is never acceptable. Recovery is declined, conservatively, when:

  • a COMMIT’s outcome is genuinely unknown — never retried. 08007, with guidance to verify. Covers COMMIT, END, PREPARE TRANSACTION and COMMIT PREPARED.
  • part of a response already reached the client — the session closes rather than concatenate a second result onto the first.
  • the transaction ran at SERIALIZABLE or REPEATABLE READ — set on BEGIN, via SET TRANSACTION, or inherited from default_transaction_isolation. No replacement backend can reproduce that snapshot, so the transaction is marked non-replayable.
  • a response was larger than [limits] tr_max_observation_bytes — it has no digest to verify against, so its transaction is not replayed on faith.
  • an interrupted read calls anything outside the side-effect-free set — 08007, and it is never run a second time.
  • tracked session state exceeded [limits] tr_max_session_set_statements — the failover is refused with 08006 rather than re-homing a session with incomplete state. The cap counts distinct variables.
  • a COPY was in progress at the fault — 08006, connection closed.
  • replay verification detected divergence — the replay is rolled back and the interrupted statement returns 40001.

Verification catches divergent results; it does not create determinism. Replay will not make now(), random() or a RETURNING serial reproduce their first values.

What is never restored

Not restored across a failover, in any build, with or without ha-tr:

  • SQL-level PREPAREd statements
  • session temporary tables
  • open cursors
  • advisory locks

SETs issued through the extended query protocol — as most JDBC drivers send them — are not tracked either, and so are not restored. SET LOCAL, SET TRANSACTION and SET CONSTRAINTS are transaction-scoped by definition and never tracked.

The src/session_migrate.rs and src/cursor_restore.rs modules implement parts of this, but they are library components and are not on the recovery path. Applications depending on that state surviving a failover still need application-level coordination.

Sequence (transaction mode)

Client                     HeliosProxy                        PostgreSQL
  │                            │                                  │
  │── BEGIN ─────────────────▶ │── BEGIN ────────────────────────▶│ primary
  │── INSERT ... ────────────▶ │── INSERT ... ───────────────────▶│
  │                            │   [record: BEGIN, INSERT         │
  │                            │    + digest of response seen]    │
  │── UPDATE ... ────────────▶ │── UPDATE ... ──────────────────  ✗ primary dies
  │                            │   [detect; wait for new primary] │
  │                            │   ←── one write_timeout_secs ──→ │
  │                            │── connect, auth, SET restore ───▶│ new primary
  │                            │── BEGIN; INSERT ...; UPDATE ... ▶│  (replayed)
  │                            │   [re-digest; compare]           │
  │                            │   match -> continue              │
  │                            │   differ -> ROLLBACK, 40001      │
  │◀─ UPDATE result ────────── │◀─ ... ───────────────────────── │
  │── COMMIT ────────────────▶ │── COMMIT ───────────────────────▶│
  │◀─ COMMIT OK ───────────── │◀─ COMMIT OK ──────────────────── │

If the failure had happened while the COMMIT itself was outstanding, the client would instead receive 08007 and nothing would be retried.

Configuration

tr_enabled = true
tr_mode = "session"        # none | session | select | transaction
write_timeout_secs = 30    # ONE deadline for the whole recovery
tr_read_functions = []     # extra provably-pure functions an interrupted read may call


[auth]
mode = "scram"             # required to fail over onto password-protected backends


[limits]
tr_max_replay_statements = 1000      # cap on statements recorded per transaction
tr_max_replay_bytes = 4194304        # cap on recorded bytes per transaction (4 MiB)
tr_max_session_set_statements = 256  # cap on DISTINCT session variables restored
tr_max_observation_bytes = 1048576   # per-statement response-digest budget (1 MiB)
max_backend_frame_bytes = 104857600  # ceiling on one backend protocol frame (100 MiB)
backend_response_timeout_secs = 0    # 0 = off; whole-response deadline

These are the shipping defaults. A transaction that exceeds a replay bound is not replayed; the client receives an error instead of a partial replay. Exceeding tr_max_session_set_statements refuses the failover outright with 08006.

tr_read_functions takes plain unqualified identifiers, matched case-insensitively; schema qualification is ignored, so pg_catalog.lower(...) is lower. A call through a quoted identifier ("MyFn"(...)) is never eligible — its case is opaque to the policy.

max_backend_frame_bytes and backend_response_timeout_secs are relay hardening rather than TR policy: the first closes a backend whose declared frame length is above the cap or below the 4-byte protocol minimum, the second bounds a backend that drips bytes inside a single response, which a per-read timeout cannot catch.

Metrics

Exported on the admin endpoint (/metrics, Prometheus format):

MetricMeaning
tr_failovers_totalPrimary failovers handled by the TR path
tr_statements_reexecuted_totalAutocommit statements and reads re-executed transparently
tr_transactions_replayed_totalUncommitted transactions replayed in transaction mode
tr_replay_failures_totalReplays that could not be completed (client got an error)
tr_unknown_outcome_errors_total08007 errors returned because an outcome was unknown
tr_replay_cap_exceeded_totalTransactions marked non-replayable by a tr_max_replay_* bound
tr_session_set_cap_exceeded_totalSessions whose SET tracking hit tr_max_session_set_statements; a subsequent failover is refused with 08006

The Admin API reference documents the same counters alongside the rest of /metrics.

Requirements and limits

  • Auth boundary. Failing over onto password-protected backends requires the proxy to be the authentication boundary: [auth] mode = "scram". In pass-through auth the proxy never sees the client’s password (SCRAM is designed for that), so a fresh backend connection can only be completed against a backend that does not challenge; a challenging backend fails the recovery immediately with 08006.
  • Session-state restore is transactional. SET/RESET tracking mirrors PostgreSQL: a SET inside a transaction takes effect for restore only when that transaction commits, ROLLBACK TO SAVEPOINT discards the ones issued after the savepoint (RELEASE keeps them), and RESET / RESET ALL / DISCARD ALL inside a transaction are deferred to commit — so a rolled-back RESET ALL no longer wipes the restore set. Variables are keyed by name, so repeating a SET reuses its slot.
  • GUC restore scope. Restore covers simple-protocol SET / RESET statements. Parameters changed by other means — inside functions, or via SET sent through the extended protocol — are not tracked. See What is never restored.
  • Snapshot-pinned transactions are never replayed. SERIALIZABLE and REPEATABLE READ are non-replayable by policy; see What it refuses to do.
  • Replay resends the original SQL. Non-deterministic statements — now(), clock_timestamp(), random(), RETURNING of serial or sequence values — can produce different results on the new primary. Verification will catch a divergent result and roll the replay back with 40001, but it cannot make the values reproduce. This is why transaction mode is opt-in: enable it for workloads whose transactions are deterministic or idempotent.
  • COPY is never resumed (08006).
  • Replay bounds are governed by the [limits] keys above.

Verification

The behaviour on this page is exercised against live PostgreSQL, not asserted:

  • 77-check failover battery (scripts/regress/tr-failover-test.sh) across all four tr_mode values against PostgreSQL 18.4 — interrupted autocommit statements, interrupted reads, uncommitted transactions, in-flight COMMITs, prepared statements and COPY. This established the base path at 1.6.0.
  • Live commit-outcome suite, 7 cases. The connection is destroyed while a COMMIT is in flight, then the suite asserts against the database’s real state whether the client’s report was correct. The pre-fix binary failed 4 of 7.
  • Live streaming suite, 13 cases. Faults during large result sets, across Flush chunks, mid-DataRow, after CommandComplete — asserting that no client ever sees a duplicated row or a second protocol completion. The pre-fix binary failed 12 of 13.
  • Synthetic boundary harness, 26 cases. Commit-boundary classification against comments, quoted and dollar-quoted text, multi-statement strings, backslash escapes, savepoint stacks and GUC cap behaviour.
  • Deterministic partial-writer. A writer that fails after every byte offset, confirming no offset can produce a duplicate committed write.
  • Full gate matrix on the release commit. clippy -D warnings across four feature profiles; 432 / 518 / 1726 / 1727 tests on default, ha-tr, all-features and all-features,postgres-topology, zero failures; MSRV 1.86 with a locked dependency graph.

The mode × fault-phase × transaction-state × statement-kind decision table (tr_decide) is exhaustively unit-tested.

Release history

  • 1.6.0 (2026-09-04) — introduced tr_mode in-session failover.
  • 1.6.1 (2026-09-09) — fixed commit-outcome, publication and delivery safety.
  • 1.7.0 (2026-09-11) — completes the series: verification, refusals, and one deadline.