How a Series B payments company replaced PostgreSQL + Elasticsearch + S3 with a single HeliosDB-Lite deployment.
A Series B payments processing company with 80 engineers was drowning in compliance infrastructure. Their platform processes over 2 million transactions daily across 14 microservices, and every state mutation -- from account creation to fund transfers to KYC status changes -- needed an immutable, tamper-evident audit trail. Federal regulators required the ability to reconstruct the exact state of any account at any point in time, with cryptographic proof that records had not been altered.
Their existing architecture had grown organically over three years. Every microservice wrote audit events to a shared PostgreSQL cluster, which replicated to Elasticsearch for search, while a custom CDC pipeline fed a separate "audit lake" in S3 for long-term retention. A team of four engineers maintained the audit pipeline full-time, dealing with replication lag, schema drift between the primary database and Elasticsearch, and periodic data reconciliation jobs that ran for hours.
The breaking point came during their first SOC 2 Type II audit. The auditors asked to see the exact state of a specific merchant account on a specific date six months prior. What should have been a simple query turned into a three-day engineering effort involving S3 data lake queries, cross-referencing Elasticsearch logs, and manual reconstruction from PostgreSQL WAL archives. The CTO decided the audit infrastructure needed a fundamental rethink.
| Problem | Impact |
|---|---|
| No native time-travel | Reconstructing historical state required replaying audit logs -- slow and error-prone |
| Three systems to maintain | PostgreSQL, Elasticsearch, S3 audit lake -- each with its own failure modes |
| Schema drift | Elasticsearch mappings frequently fell out of sync with PostgreSQL schemas |
| No tamper evidence | Audit records in PostgreSQL could theoretically be modified by DBAs |
| Replication lag | Up to 30 seconds between PostgreSQL write and Elasticsearch availability |
| High cost | $14K/month for Elasticsearch cluster alone, plus $3K/month for S3 + Glue |
| 4 FTE maintenance burden | Dedicated team just to keep audit pipeline running |
| Compliance query latency | Average 8.2 seconds for historical state reconstruction |
The team replaced their entire audit infrastructure with HeliosDB-Lite, leveraging three key capabilities: built-in time-travel queries, transparent data encryption (TDE) with zero-knowledge encryption (ZKE), and native full-text search.
Each microservice embeds HeliosDB-Lite directly, writing both operational data and audit events to a single database with TDE enabled. Time-travel queries replace the need for custom audit log reconstruction. A central compliance service queries any microservice's database for historical state.
Connection setup via the PostgreSQL wire protocol:
-- Enable TDE with AES-256-GCM (done once at database initialization)
-- All data at rest is encrypted transparently
SET helios.encryption = 'tde';
SET helios.encryption_provider = 'aws-lc';
-- Create the transactions table with standard SQL
CREATE TABLE transactions (
id BIGINT PRIMARY KEY,
merchant_id INT NOT NULL,
amount DECIMAL(12, 2) NOT NULL,
currency VARCHAR(3) NOT NULL,
status VARCHAR(20) NOT NULL,
customer_email TEXT,
metadata JSONB,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_transactions_merchant ON transactions (merchant_id);
CREATE INDEX idx_transactions_status ON transactions (status);The core capability that eliminated three months of audit preparation:
-- "Show me the exact state of merchant 4471's account on January 15th"
-- This is what took 3 days before. Now it's a single query.
SELECT *
FROM transactions
AS OF TIMESTAMP '2025-01-15T23:59:59Z'
WHERE merchant_id = 4471;
-- "Show me every change to this transaction over its lifetime"
SELECT *
FROM transactions
VERSIONS BETWEEN TIMESTAMP '2025-01-01' AND '2025-06-30'
WHERE id = 889201
ORDER BY helios_version_timestamp;
-- "Compare account balances between two dates for reconciliation"
WITH jan_state AS (
SELECT merchant_id, SUM(amount) as total
FROM transactions
AS OF TIMESTAMP '2025-01-31T23:59:59Z'
WHERE status = 'settled'
GROUP BY merchant_id
),
feb_state AS (
SELECT merchant_id, SUM(amount) as total
FROM transactions
AS OF TIMESTAMP '2025-02-28T23:59:59Z'
WHERE status = 'settled'
GROUP BY merchant_id
)
SELECT
j.merchant_id,
j.total AS jan_total,
f.total AS feb_total,
f.total - j.total AS net_change
FROM jan_state j
JOIN feb_state f ON j.merchant_id = f.merchant_id
ORDER BY net_change DESC;import psycopg2
from datetime import datetime, timedelta
class ComplianceReporter:
"""Connects to HeliosDB-Lite via standard PostgreSQL wire protocol."""
def __init__(self, host: str = "localhost", port: int = 5433):
self.conn = psycopg2.connect(
host=host,
port=port,
dbname="payments",
user="compliance_reader",
password="<from-vault>",
sslmode="require"
)
def reconstruct_account_state(
self, merchant_id: int, as_of: datetime
) -> dict:
"""
Reconstruct the exact state of a merchant account at a point in time.
Previously required 3 days of engineering effort across 3 systems.
Now executes in under 200ms.
"""
with self.conn.cursor() as cur:
cur.execute("""
SELECT
merchant_id,
COUNT(*) as transaction_count,
SUM(CASE WHEN status = 'settled' THEN amount ELSE 0 END) as settled_total,
SUM(CASE WHEN status = 'pending' THEN amount ELSE 0 END) as pending_total,
SUM(CASE WHEN status = 'refunded' THEN amount ELSE 0 END) as refunded_total,
MAX(updated_at) as last_activity
FROM transactions
AS OF TIMESTAMP %s
WHERE merchant_id = %s
GROUP BY merchant_id
""", (as_of.isoformat(), merchant_id))
row = cur.fetchone()
if not row:
return {"merchant_id": merchant_id, "status": "no_records"}
return {
"merchant_id": row[0],
"as_of": as_of.isoformat(),
"transaction_count": row[1],
"settled_total": float(row[2]),
"pending_total": float(row[3]),
"refunded_total": float(row[4]),
"last_activity": row[5].isoformat() if row[5] else None,
"encryption": "AES-256-GCM",
"tamper_evident": True
}
def generate_audit_report(
self, start_date: datetime, end_date: datetime
) -> list[dict]:
"""
Generate a full audit trail for a date range.
Uses time-travel VERSIONS BETWEEN to capture every state change.
"""
with self.conn.cursor() as cur:
cur.execute("""
SELECT
id,
merchant_id,
amount,
status,
metadata,
helios_version_timestamp,
helios_version_operation
FROM transactions
VERSIONS BETWEEN TIMESTAMP %s AND %s
ORDER BY helios_version_timestamp ASC
""", (start_date.isoformat(), end_date.isoformat()))
return [
{
"transaction_id": row[0],
"merchant_id": row[1],
"amount": float(row[2]),
"status": row[3],
"metadata": row[4],
"changed_at": row[5].isoformat(),
"operation": row[6]
}
for row in cur.fetchall()
]
# Usage during SOC 2 audit
reporter = ComplianceReporter()
# Auditor request: "Show merchant 4471's state on Jan 15"
state = reporter.reconstruct_account_state(4471, datetime(2025, 1, 15, 23, 59, 59))
print(f"Settled total: ${state['settled_total']:,.2f}")
print(f"Encryption: {state['encryption']}")
# Auditor request: "Full audit trail for Q1"
trail = reporter.generate_audit_report(
datetime(2025, 1, 1),
datetime(2025, 3, 31, 23, 59, 59)
)
print(f"Total state changes in Q1: {len(trail)}")Replacing Elasticsearch for audit log search:
-- Search transaction metadata for specific patterns
-- (previously required Elasticsearch with 30s replication lag)
SELECT id, merchant_id, amount, status, metadata
FROM transactions
WHERE metadata @@ 'chargeback AND disputed'
ORDER BY updated_at DESC
LIMIT 50;
-- Combine time-travel with full-text search:
-- "Find all disputed transactions for merchant 4471 as they existed on March 1st"
SELECT id, amount, status, metadata
FROM transactions
AS OF TIMESTAMP '2025-03-01T00:00:00Z'
WHERE merchant_id = 4471
AND metadata @@ 'disputed'
ORDER BY created_at DESC;| Metric | Before (PG + ES + S3) | After (HeliosDB-Lite) | Improvement |
|---|---|---|---|
| Monthly infrastructure cost | $17,200 | $860 | 95% reduction |
| Audit query latency (P50) | 8.2s | 180ms | 97.8% faster |
| Audit query latency (P95) | 34s | 620ms | 98.2% faster |
| Compliance query latency | 3 days (manual) | 3.1s (automated) | From days to seconds |
| SOC 2 preparation time | 3 months | 2 weeks | 83% faster |
| Systems to maintain | 3 (PG + ES + S3) | 1 (HeliosDB-Lite) | 67% fewer |
| Dedicated audit engineers | 4 FTE | 0.5 FTE | 87.5% reduction |
| Replication lag | Up to 30s | 0 (single system) | Eliminated |
| Data encrypted at rest | Partial (S3 only) | 100% (TDE + ZKE) | Full coverage |
| Tamper evidence | None | Built-in (MVCC versioning) | New capability |
AS OF TIMESTAMP. The immutable MVCC versioning model means every historical state is preserved automatically -- no CDC pipelines, no audit tables, no reconstruction logic.| Component | Technology |
|---|---|
| Database | HeliosDB-Lite (embedded, single binary) |
| Encryption | TDE (AES-256-GCM) + ZKE via AWS-LC FIPS provider |
| Client Protocol | PostgreSQL wire protocol (SCRAM-SHA-256 auth) |
| Application Language | Python 3.12 (compliance service), Go 1.22 (microservices) |
| Driver | psycopg2 (Python), pgx (Go) |
| Deployment | Kubernetes (GKE), HeliosDB-Lite embedded in each pod |
| Key Management | AWS KMS (master key), HeliosDB-Lite (data encryption keys) |
| Monitoring | Prometheus + Grafana (standard PostgreSQL metrics exporters) |
| Previous Stack (replaced) | PostgreSQL 15, Elasticsearch 8.x, S3 + AWS Glue, custom CDC pipeline |
Schedule a demo to see how HeliosDB can transform your data infrastructure.