--- title: "Transaction Replay (TR) — In-Session Failover" --- # Transaction Replay (TR) — In-Session Failover Transaction Replay keeps client sessions alive through a primary failure: sessions re-home and restore state, an interrupted read re-runs when it is provably side-effect-free, and opt-in transaction mode replays uncommitted work on the new primary — verifying every replayed response against what the client already saw, and never retrying a COMMIT whose outcome is unknown. **Transaction Replay is part of the default build.** It needs no cargo feature and no application change — any PostgreSQL-wire client, any PG-wire backend. Every part of it is always compiled: in-session recovery, the recovery journal and the operator-driven `/api/replay` and `/api/shadow` endpoints. `tr_enabled` is the runtime master switch: `tr_enabled = false` (or `--tr=false` / `--tr false` on the command line) stops journaling, forces the effective `tr_mode` to `none`, and makes `POST /api/replay` return `503`. The `ha-tr` [feature](/docs/proxy/feature-flags/) is a deprecated no-op. The journal records real committed transactions and can be made durable on disk; see [Recovery journal and committed-history replay](#recovery-journal-and-committed-history-replay). In-session replay verifies its own results. This page states exactly what each mode does, what is always true regardless of mode, what the proxy refuses to do, and the requirements and limits you must plan for. It does not claim zero data loss for in-flight COMMITs — see [What is always true](#what-is-always-true) — and it does not claim cursor, temp-table or `PREPARE` survival; see [What is never restored](#what-is-never-restored). ## Why it matters In a plain PostgreSQL HA setup, when the primary fails the pool drops active connections, in-flight statements get a connection-reset error, and every application must reconnect and decide on its own whether to retry. Most do not do this correctly. With Transaction Replay the proxy is the reconnect-and-retry layer: the client connection stays open, the proxy re-homes it to the new primary, restores the session state, and — depending on `tr_mode` — re-executes what can safely be re-executed. Anything whose outcome cannot be established is reported to the client with a precise SQLSTATE instead of being guessed. ## In-session replay (the core path) This is what a live client experiences when its backend dies mid-session. It is independent of the recovery journal and the administrative replay endpoints described further down. **What is recorded.** While a session is inside a transaction under `tr_mode = select` or `transaction`, the proxy records each statement, the session state it changed, and a bounded digest of the response frames the client was shown (`[limits] tr_max_observation_bytes`, default 1 MiB). Autocommit traffic records nothing. **What happens on a backend fault.** The proxy classifies how far the statement got. If the outcome is *not delivered*, recovery is safe. If the outcome is *unknown* — the statement may have committed — it is never re-executed. Once any part of a response has reached the client, the session is closed rather than have a second result appended to the first. **What recovery does.** Under one deadline (`write_timeout_secs`) it waits for a healthy primary, connects and authenticates, restores the session's tracked `SET` state, and replays the recorded transaction. Each replayed statement's response is hashed and compared with what the client originally saw; any divergence rolls the replay back and returns `40001`. ## Behaviour by mode (`tr_mode`) | `tr_mode` | What the client sees when the primary fails | |---|---| | `none` | A proper PostgreSQL error (SQLSTATE `57P01`, admin shutdown) and the connection closes. Equivalent to a plain pooler. | | `session` *(default)* | The client connection stays alive. The proxy re-homes it to the new primary and restores `SET`/GUC state. The interrupted statement returns **one** error: `57P01` if it never reached the old primary, `08007` (transaction resolution unknown) if its outcome is unknown. Autocommit statements that had not yet been delivered are re-executed transparently. | | `select` | Everything in `session`, plus an interrupted **read** is re-run on the new primary — **but only when every function it calls is provably side-effect-free** (a PostgreSQL built-in on the proxy's allowlist, or a name you listed in `tr_read_functions`). A read calling a user-defined function, `nextval`, `pg_notify`, `set_config`, an advisory lock, a quoted-identifier call, or using `SELECT … INTO` is opaque: it returns `08007` and is never run twice. Read-only explicit transactions are replayed from `BEGIN`. | | `transaction` *(opt-in)* | Everything in `select`, plus the **uncommitted transaction** is replayed from `BEGIN` on the new primary and the in-flight statement re-executed inside it. Replay is **verified**: each statement's original response frames were digested as the client saw them (within `[limits] tr_max_observation_bytes`), the replacement backend's responses are digested the same way, and any divergence rolls the replay back and returns `40001` rather than continuing on top of rows the client never observed. A failed replayed statement produces the same `40001`. | ## What is always true These hold in every mode, including `transaction`: - **A COMMIT whose outcome is unknown is never retried.** The client receives `08007` and must check before retrying. This is what prevents double-apply; it also means Transaction Replay does not promise zero data loss for a COMMIT that was in flight at the moment of failure. - **One deadline covers the whole recovery.** `write_timeout_secs` bounds waiting for a healthy primary, connect and authentication, session-state restore, replay and re-execution together, so the phases cannot add up past the configured value. If no primary appears in the window the client gets `08006`. - **One error, then a clean transaction state.** When the single error is returned inside an explicit transaction, the client-visible transaction is aborted (`25P02` until the client issues `ROLLBACK`), exactly as PostgreSQL would. - **A replay that fails yields `40001`** (serialization failure), so standard retry logic applies; the session is not left half-replayed. The same `40001` is returned when replay verification detects divergence. - **Protocol-level named prepared statements are re-prepared** lazily on the new primary as they are next used. This is the extended-protocol `Parse`-with-a-name case only — SQL-level `PREPARE foo AS …` is **not** restored. See [What is never restored](#what-is-never-restored). - **A COPY in progress always fails** with `08006` (connection failure). COPY streams are not journaled or resumed. - The proxy returns real PostgreSQL SQLSTATEs (`57P01`, `08007`, `08006`, `40001`) so existing driver and application retry logic keeps working. ## What it refuses to do Transparent recovery is the goal, a clear error is the fallback, and a wrong answer is never acceptable. Recovery is declined, conservatively, when: - **a `COMMIT`'s outcome is genuinely unknown** — never retried. `08007`, with guidance to verify. Covers `COMMIT`, `END`, `PREPARE TRANSACTION` and `COMMIT PREPARED`. - **part of a response already reached the client** — the session closes rather than concatenate a second result onto the first. - **the transaction ran at `SERIALIZABLE` or `REPEATABLE READ`** — set on `BEGIN`, via `SET TRANSACTION`, or inherited from `default_transaction_isolation`. No replacement backend can reproduce that snapshot, so the transaction is marked non-replayable. - **a response was larger than `[limits] tr_max_observation_bytes`** — it has no digest to verify against, so its transaction is not replayed on faith. - **an interrupted read calls anything outside the side-effect-free set** — `08007`, and it is never run a second time. - **tracked session state exceeded `[limits] tr_max_session_set_statements`** — the failover is refused with `08006` rather than re-homing a session with incomplete state. The cap counts *distinct* variables. - **a `COPY` was in progress at the fault** — `08006`, connection closed. - **replay verification detected divergence** — the replay is rolled back and the interrupted statement returns `40001`. Verification catches divergent *results*; it does not create determinism. Replay will not make `now()`, `random()` or a `RETURNING` serial reproduce their first values. ## What is never restored Not restored across a failover, in any build: - SQL-level `PREPARE`d statements - session temporary tables - open cursors - advisory locks `SET`s issued through the **extended query protocol** — as most JDBC drivers send them — are not tracked either, and so are not restored. `SET LOCAL`, `SET TRANSACTION` and `SET CONSTRAINTS` are transaction-scoped by definition and never tracked. The `src/session_migrate.rs` and `src/cursor_restore.rs` modules implement parts of this, but they are library components and are **not** on the recovery path. Applications depending on that state surviving a failover still need application-level coordination. ## Sequence (transaction mode) ``` Client HeliosProxy PostgreSQL │ │ │ │── BEGIN ─────────────────▶ │── BEGIN ────────────────────────▶│ primary │── INSERT ... ────────────▶ │── INSERT ... ───────────────────▶│ │ │ [record: BEGIN, INSERT │ │ │ + digest of response seen] │ │── UPDATE ... ────────────▶ │── UPDATE ... ────────────────── ✗ primary dies │ │ [detect; wait for new primary] │ │ │ ←── one write_timeout_secs ──→ │ │ │── connect, auth, SET restore ───▶│ new primary │ │── BEGIN; INSERT ...; UPDATE ... ▶│ (replayed) │ │ [re-digest; compare] │ │ │ match -> continue │ │ │ differ -> ROLLBACK, 40001 │ │◀─ UPDATE result ────────── │◀─ ... ───────────────────────── │ │── COMMIT ────────────────▶ │── COMMIT ───────────────────────▶│ │◀─ COMMIT OK ───────────── │◀─ COMMIT OK ──────────────────── │ ``` If the failure had happened while the **COMMIT** itself was outstanding, the client would instead receive `08007` and nothing would be retried. ## Configuration ```toml tr_enabled = true tr_mode = "session" # none | session | select | transaction write_timeout_secs = 30 # ONE deadline for the whole recovery tr_read_functions = [] # extra provably-pure functions an interrupted read may call [auth] mode = "scram" # required to fail over onto password-protected backends [limits] tr_max_replay_statements = 1000 # cap on statements recorded per transaction tr_max_replay_bytes = 4194304 # cap on recorded bytes per transaction (4 MiB) tr_max_session_set_statements = 256 # cap on DISTINCT session variables restored tr_max_observation_bytes = 1048576 # per-statement response-digest budget (1 MiB) max_backend_frame_bytes = 104857600 # ceiling on one backend protocol frame (100 MiB) backend_response_timeout_secs = 0 # 0 = off; whole-response deadline ``` These are the defaults. A transaction that exceeds a replay bound is not replayed; the client receives an error instead of a partial replay. Exceeding `tr_max_session_set_statements` refuses the failover outright with `08006`. `tr_read_functions` takes plain unqualified identifiers, matched case-insensitively; schema qualification is ignored, so `pg_catalog.lower(...)` is `lower`. A call through a quoted identifier (`"MyFn"(...)`) is never eligible — its case is opaque to the policy. `max_backend_frame_bytes` and `backend_response_timeout_secs` are relay hardening rather than TR policy: the first closes a backend whose declared frame length is above the cap or below the 4-byte protocol minimum, the second bounds a backend that drips bytes inside a single response, which a per-read timeout cannot catch. ## Metrics Exported on the admin endpoint (`/metrics`, Prometheus format): | Metric | Meaning | |---|---| | `tr_failovers_total` | Primary failovers handled by the TR path | | `tr_statements_reexecuted_total` | Autocommit statements and reads re-executed transparently | | `tr_transactions_replayed_total` | Uncommitted transactions replayed in `transaction` mode | | `tr_replay_failures_total` | Replays that could not be completed (client got an error) | | `tr_unknown_outcome_errors_total` | `08007` errors returned because an outcome was unknown | | `tr_replay_cap_exceeded_total` | Transactions marked non-replayable by a `tr_max_replay_*` bound | | `tr_session_set_cap_exceeded_total` | Sessions whose `SET` tracking hit `tr_max_session_set_statements`; a subsequent failover is refused with `08006` | The [Admin API reference](/docs/proxy/admin-api/) documents the same counters alongside the rest of `/metrics`. ## Requirements and limits - **Auth boundary.** Failing over onto password-protected backends requires the proxy to be the authentication boundary: `[auth] mode = "scram"`. In pass-through auth the proxy never sees the client's password (SCRAM is designed for that), so a fresh backend connection can only be completed against a backend that does not challenge; a challenging backend fails the recovery immediately with `08006`. - **Session-state restore is transactional.** `SET`/`RESET` tracking mirrors PostgreSQL: a `SET` inside a transaction takes effect for restore only when that transaction commits, `ROLLBACK TO SAVEPOINT` discards the ones issued after the savepoint (`RELEASE` keeps them), and `RESET` / `RESET ALL` / `DISCARD ALL` inside a transaction are deferred to commit — so a rolled-back `RESET ALL` no longer wipes the restore set. Variables are keyed by name, so repeating a `SET` reuses its slot. - **GUC restore scope.** Restore covers simple-protocol `SET` / `RESET` statements. Parameters changed by other means — inside functions, or via `SET` sent through the extended protocol — are not tracked. See [What is never restored](#what-is-never-restored). - **Snapshot-pinned transactions are never replayed.** `SERIALIZABLE` and `REPEATABLE READ` are non-replayable by policy; see [What it refuses to do](#what-it-refuses-to-do). - **Replay resends the original SQL.** Non-deterministic statements — `now()`, `clock_timestamp()`, `random()`, `RETURNING` of serial or sequence values — can produce different results on the new primary. Verification will catch a *divergent result* and roll the replay back with `40001`, but it cannot make the values reproduce. This is why `transaction` mode is opt-in: enable it for workloads whose transactions are deterministic or idempotent. - **COPY** is never resumed (`08006`). - Replay bounds are governed by the `[limits]` keys above. ## Verification The behaviour on this page is exercised against live PostgreSQL, not asserted: - **77-check failover battery** (`scripts/regress/tr-failover-test.sh`) across all four `tr_mode` values — interrupted autocommit statements, interrupted reads, uncommitted transactions, in-flight COMMITs, prepared statements and COPY (internal tests, HeliosProxy 1.6.0 against PostgreSQL 18.4). - **Live commit-outcome suite, 7 cases.** The connection is destroyed while a `COMMIT` is in flight, then the suite asserts against the database's real state whether the client's report was correct. The pre-fix binary failed 4 of 7. - **Live streaming suite, 13 cases.** Faults during large result sets, across `Flush` chunks, mid-`DataRow`, after `CommandComplete` — asserting that no client ever sees a duplicated row or a second protocol completion. The pre-fix binary failed 12 of 13. - **Synthetic boundary harness, 26 cases.** Commit-boundary classification against comments, quoted and dollar-quoted text, multi-statement strings, backslash escapes, savepoint stacks and GUC cap behaviour. - **Deterministic partial-writer.** A writer that fails after every byte offset, confirming no offset can produce a duplicate committed write. - **Full gate matrix.** `clippy -D warnings` across four feature profiles; 432 / 518 / 1726 / 1727 tests on `default`, `ha-tr`, `all-features` and `all-features,postgres-topology`, zero failures; MSRV 1.86 with a locked dependency graph (internal tests, HeliosProxy 1.7.0 release commit). The mode × fault-phase × transaction-state × statement-kind decision table (`tr_decide`) is exhaustively unit-tested. ## Recovery journal and committed-history replay Separate from in-session recovery, the proxy keeps a **recovery journal** of the transactions it saw commit. It feeds the operator endpoint `POST /api/replay` and the commit-aware [query cache](/docs/proxy/configuration/#query-cache-cache) invalidation (the capture also runs when `tr_enabled = false` but the cache is on). ### What the journal records The proxy registers what it sends — simple-query strings, and extended-protocol `Parse`/`Bind`/`Execute` with every bound parameter value byte for byte in its text or binary format — and reconciles it with what the backend answered: each `CommandComplete` tag, the first `ErrorResponse`, and the `ReadyForQuery` transaction status. So the journal holds **real transactions**: - `BEGIN … COMMIT` boundaries as the backend reported them, with per-statement outcomes and `SAVEPOINT` / `ROLLBACK TO` structure (a rollback to a savepoint drops the statements after it). A transaction enters committed history only when the backend's closing tag was `COMMIT`; a `COMMIT` in an aborted transaction answers `ROLLBACK` and is treated as one. Auto-commit statements become one committed transaction each. - Rejected statements, rolled-back transactions, sessions that end mid-transaction and two-phase transactions (`PREPARE TRANSACTION`) never enter committed history. - Each committed transaction carries its source identity (client address, user, database, backend node, tenant) and a **global commit sequence** (`commit_seq`) in the order this proxy observed the commits. - Reads are not journaled, so a read with side effects (`nextval`, `set_config`, a volatile function) is not reproduced. A `COPY … FROM STDIN`, an `EXECUTE` of a session-scoped prepared statement, a statement over `[journal] max_statement_bytes`, or a transaction over a per-transaction cap is kept but marked **incomplete**: committed-history replay refuses it rather than apply it partially. ### Retention and durability (`[journal]`) ```toml [journal] max_committed_transactions = 50000 # kept in memory, and reloaded from dir at startup max_committed_bytes = 268435456 # 256 MiB in memory dir = "/var/lib/heliosproxy/journal" # empty (default) = memory only segment_bytes = 67108864 # rotate segments at 64 MiB retain_bytes = 1073741824 # keep at most 1 GiB on disk fsync = "interval" # commit | interval | none fsync_interval_ms = 100 writer_queue = 4096 ``` - **Without `dir`** the journal lives in process memory, bounded by `max_committed_transactions` / `max_committed_bytes`; nothing survives a restart. - **With `dir`** every committed transaction is also appended to `journal-.log` segments: CRC-checked, postcard-encoded records (format `HJ02`), rotated at `segment_bytes`, with the oldest whole segments deleted beyond `retain_bytes`. At startup the segments are read back, the newest `max_committed_transactions` are reloaded, and `commit_seq` numbering continues after the highest recovered value. An unwritable directory fails startup. - **Torn tails.** If the proxy crashed mid-append, the segment's torn or corrupt tail is truncated at the last intact record on startup instead of failing the load. (A directory in the older pre-release format `HJ01`, which no released version wrote, is refused with an explicit error rather than truncated.) - **`fsync`**: `"commit"` runs `fdatasync` after every record; `"interval"` (default) groups an `fdatasync` every `fsync_interval_ms`; `"none"` leaves it to the OS page cache. Under every policy the data path never waits on disk: records go through a bounded queue (`writer_queue`) to a writer thread, and a full queue drops the record and counts it (`coverage.dropped_transactions` on replay responses). - **What it is not.** A record is written *after* the backend reported the commit — the proxy is not a participant in the backend's commit — so a crash between the two loses that record. The journal is a faithful log of what this proxy observed, not a WAL the database waits on. The commit order is a total order over everything routed through this proxy; transactions that bypassed it are not in it. - **Protect the directory.** Segments hold statement text and bound parameter values of committed writes, unencrypted — treat `dir` like the database's own data files. All `[journal]` keys and their defaults are in the [Configuration reference](/docs/proxy/configuration/#recovery-journal-journal). ### Operator replay: `POST /api/replay` Two modes ([request and response shape](/docs/proxy/admin-api/#post-apireplay)): - **`time_window`** (default, best effort) — the statements of every committed transaction in a `[from, to]` window, flattened in timestamp order and re-executed one by one on a target connection; failures are counted, not fatal (`partial: true`). Use it to hydrate a staging copy. It reads committed history only, so failed or rolled-back statements are never replayed. - **`committed_history`** (recovery grade) — the transactions whose *commit* was observed in the window (optionally only those after `after_commit_seq`), applied in commit order, one transaction at a time, each in its own `BEGIN … COMMIT` on one connection, with parameters re-sent in their captured text/binary format through the extended protocol. The first failure rolls that transaction back and **stops** (`partial: true`, `stopped_at`), so the target holds exactly the transactions up to `last_commit_seq` and nothing partial; pass that value as `after_commit_seq` to resume. Incomplete transactions are refused before they start. Both modes are bounded by `[limits] replay_deadline_secs` (default 300) and return a `coverage` block — retained counts and bytes, `commit_seq_high`, `dropped_transactions`, and whether the run survives a restart (`true` when `[journal] dir` is set) — so a response always says what backed it. Replay re-executes SQL; it does not copy rows, so `now()`, `random()` and sequence values produce fresh values on the target. ## Related - [Configuration reference](/docs/proxy/configuration/) — `tr_enabled`, `tr_mode`, `tr_read_functions`, `[limits]`, `[journal]`, `[auth]` - [Feature flags](/docs/proxy/feature-flags/) — `ha-tr` is a deprecated no-op - [Architecture](/docs/proxy/architecture/) — journal and failover controller - [Admin API](/docs/proxy/admin-api/) — `/metrics` TR counters, `/api/replay`, `/api/shadow` - [Release history](/docs/proxy/releases/) — how Transaction Replay changed from release to release - Demos: [Bank Ledger](/docs/proxy/tutorials/bank-ledger/), [vs PgBouncer](/docs/proxy/tutorials/vs-pgbouncer/)