HTAP Dual-Format Storage

Storage Modes

Per-Column Storage Modes

HeliosDB provides four storage modes that can be assigned independently to each column in a table. This per-column granularity — rather than per-table — lets you optimize the physical layout for the exact access pattern each column experiences. Mix row-oriented, columnar, dictionary-encoded, and content-addressed storage within a single table.

• Default (row-oriented) — The standard row store, optimal for OLTP point lookups and transactional updates. Each row is stored as a contiguous tuple. This is the default when no STORAGE clause is specified.
• Columnar (batched analytics) — Values are stored in 1,024-row batches organized by column, enabling sequential reads and SIMD-accelerated aggregation. Ideal for columns frequently used in GROUP BY, SUM, AVG, and full-table scans.
• Dictionary (low-cardinality) — Values are replaced by compact integer codes that reference a shared dictionary. Best for columns with few distinct values such as status, category, or country — delivers both compression and fast equality predicates.
• ContentAddressed (deduplication) — Values are stored by content hash and deduplicated automatically. Ideal for large text payloads, JSON documents, or any column where many rows share identical values.

Use ALTER TABLE ... ALTER COLUMN ... SET STORAGE to migrate columns between storage modes at any time without rewriting the entire table. The EXPLAIN (STORAGE) command verifies the current layout.

-- Per-column storage modes in CREATE TABLE
CREATE TABLE events (
    id         INT PRIMARY KEY,              -- Default (row store)
    user_id    INT,                          -- Default (row store)
    category   TEXT STORAGE DICTIONARY,      -- Dictionary encoding
    amount     FLOAT8 STORAGE COLUMNAR,      -- Columnar batches
    timestamp  INT8 STORAGE COLUMNAR,        -- Columnar batches
    payload    TEXT STORAGE CONTENT_ADDRESSED -- Deduplication
);

-- Migrate columns between storage modes
ALTER TABLE events
  ALTER COLUMN user_id SET STORAGE COLUMNAR;

-- Verify storage layout
EXPLAIN (STORAGE) SELECT * FROM events;

Query Execution

Columnar Scan Detection

When all columns in a scanned table use STORAGE COLUMNAR, the EXPLAIN output shows “Columnar Scan” instead of the standard “Seq Scan”. This confirms that the query engine is using the batch columnar scan, which reads column batches sequentially for analytics throughput.

• Automatic detection — No hints or configuration required. The planner inspects column storage modes and selects the columnar scan operator when all referenced columns are columnar.
• Batch processing — The columnar scan reads data in 1,024-row batches, enabling SIMD-accelerated predicate evaluation and vectorized aggregation.
• I/O efficiency — Sequential column reads minimize random I/O. Only the columns referenced in the query are read from storage, which cuts the bytes read on wide tables.
• Mixed-mode fallback — If any referenced column uses a non-columnar storage mode, the standard row scan is used and resolves the column-store references transparently. The query always returns correct results regardless of storage layout.

-- All columns use STORAGE COLUMNAR
CREATE TABLE metrics (
    ts    INT8 STORAGE COLUMNAR,
    value FLOAT8 STORAGE COLUMNAR
);

helios> EXPLAIN ANALYZE SELECT * FROM metrics;
-- Shows: → Columnar Scan metrics
-- Instead of: → Scan metrics

-- The columnar scan reads column
-- batches sequentially; aggregations
-- read only the referenced columns:

helios> EXPLAIN ANALYZE
        SELECT AVG(value) FROM metrics
        WHERE ts > 1704067200;
-- → Columnar Scan metrics
--      Filter: ts > 1704067200
--      Rows: 50000

System Views

HTAP Layout Statistics

Two system views provide real-time HTAP monitoring. The pg_htap_layout_stats view shows per-table storage layout information, giving you a high-level overview of how data is distributed between the row store and column store across all tables in the database.

Column	Type	Description
table_name	TEXT	Table name
row_store_size	INT8	Estimated row store size in bytes
column_store_size	INT8	Estimated column store size in bytes
row_store_rows	INT8	Number of rows in row store
column_store_rows	INT8	Number of rows in columnar batches
sync_status	TEXT	row_only, columnar_only, dual_format, or mixed
implemented	BOOLEAN	Whether HTAP is active

SELECT * FROM pg_htap_layout_stats;

-- Example output:
--  table_name | row_store_size | column_store_size
-- -----------+----------------+------------------
--  orders     |          64000 |             48000
--  users      |          12800 |                 0
--
--  row_store_rows | column_store_rows | sync_status
-- ---------------+-------------------+------------
--           1000 |              1000 | dual_format
--            200 |                 0 | row_only
--
--  implemented
-- -------------
--  true
--  true

-- Filter for dual-format tables only
SELECT table_name,
       row_store_rows,
       column_store_rows,
       sync_status
FROM pg_htap_layout_stats
WHERE sync_status = 'dual_format';

-- Monitor storage distribution
SELECT table_name,
       row_store_size AS row_bytes,
       column_store_size AS col_bytes,
       ROUND(column_store_size::float
         / (row_store_size + column_store_size)
         * 100, 1) AS columnar_pct
FROM pg_htap_layout_stats
WHERE implemented = true;

Resource Management

Resource Budgets

The HTAP resource manager accounts each query against separate OLTP and OLAP budgets so that analytical scans can be kept apart from latency-sensitive transactions. The split uses built-in defaults today (60% OLTP / 40% OLAP); enforcing the budgets as isolation and configuring them are planned.

• Automatic classification — the workload classifier assigns each query to the OLTP or OLAP budget. No application changes are needed.
• Concurrency and memory accounting — each budget tracks concurrent operations (defaults: 1,000 OLTP, 16 OLAP) and estimated memory (a 60/40 split of the built-in budget).
• Planned — enforcing the budgets, CPU scheduling and I/O prioritisation per workload, and configuring the split.

# Built-in HTAP budgets (no configuration
# keys today; making them configurable is
# planned)

OLTP budget
  share:            60%
  concurrent ops:   1000

OLAP budget
  share:            40%
  concurrent ops:   16

# Each query is classified (see EXPLAIN
# ANALYZE) and accounted against its
# budget; enforcement is planned.