Skip to content
HeliosDB HeliosDB Full

Lakehouse Unification

Lakehouse Unification — Iceberg + Delta + Relational in One SQL

Crate: heliosdb-lakehouse with sub-crates iceberg, delta, archival, catalog-unified Status: Iceberg & Delta production; Apache Hudi at 20% per main crate README; Unified API at 10% with LakehouseManager + SqlInterface ARR: $25M (crate doc) / $50M-$75M (audit aggregate)

UVP

Most lakehouses make you pick a format and live with the choice. The Full edition ships a unified lakehouse layer — one LakehouseManager, one SQL surface, three open table formats (Apache Iceberg, Delta Lake, Apache Hudi) plus your relational HeliosDB tables, all in the same query. Format auto-detected at registration. Time travel works the same way across all of them. JOIN an Iceberg customers table to a Delta orders table to a relational users table in one statement, against S3, Azure, GCS, or local.

Prerequisites

  • HeliosDB Full v7.x+.
  • Object storage (S3 / Azure / GCS / local) with at least one Iceberg or Delta table.
  • About 25 minutes.

1. The Unified Surface

From the main lakehouse README:

use heliosdb_lakehouse::{LakehouseManager, TableFormat};


#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let manager = LakehouseManager::new().await?;


    // Format auto-detected from path layout
    manager.register_table("orders", "s3://bucket/lakehouse/orders").await?;


    let results = manager.query_time_travel(
        "orders",
        Some(chrono::Utc::now() - chrono::Duration::days(7)),
        None
    ).await?;


    Ok(())
}

register_table sniffs the path:

  • _metadata/ directory → Iceberg.
  • _delta_log/ directory → Delta.
  • .hoodie/ directory → Hudi.

You don’t tell it the format. It looks.

2. SQL Interface — CREATE EXTERNAL TABLE

use heliosdb_lakehouse::*;
use std::sync::Arc;


let manager = Arc::new(LakehouseManager::new().await?);
let sql = SqlInterface::new(manager);


sql.execute(
    "CREATE EXTERNAL TABLE orders
     USING delta
     LOCATION 's3://bucket/orders'"
).await?;


sql.execute(
    "CREATE EXTERNAL TABLE IF NOT EXISTS customers
     USING iceberg
     LOCATION 's3://bucket/customers'"
).await?;


sql.execute(
    "CREATE EXTERNAL TABLE products
     USING hudi
     LOCATION 's3://bucket/products'"
).await?;

All three are now standard relational tables from the perspective of any HeliosDB SQL session.

3. JOIN Across Formats

This is the headline:

SELECT
    c.region,
    p.category,
    SUM(o.amount) AS revenue,
    COUNT(*) AS orders
FROM orders        o   -- Delta on S3
JOIN customers    c ON c.id = o.customer_id  -- Iceberg on GCS
JOIN products     p ON p.id = o.product_id   -- Hudi on Azure
JOIN users        u ON u.id = c.user_id      -- relational, in HeliosDB
WHERE o.order_date >= '2024-01-01'
GROUP BY c.region, p.category
ORDER BY revenue DESC;

The query planner pushes filters into each format adapter, parallel-fetches from each storage backend, then joins in HeliosDB’s executor. No Spark cluster, no Trino, no Presto — just the database server.

4. Time Travel — Same Three APIs Across Formats

Each format has its own native time-travel mechanic. The unified API exposes them with one call:

// By timestamp (works for Iceberg, Delta, Hudi)
let yesterday = chrono::Utc::now() - chrono::Duration::days(1);
let snapshot = manager.query_time_travel("orders", Some(yesterday), None).await?;


// By version (Delta version number, Iceberg snapshot id, Hudi commit time)
let v5 = manager.query_time_travel("orders", None, Some("5".to_string())).await?;

In SQL:

SELECT * FROM orders TIMESTAMP AS OF '2024-01-01 12:00:00';
SELECT * FROM orders VERSION AS OF 5;

5. Iceberg-Specific Operations

From heliosdb-lakehouse/crates/iceberg:

Apache Iceberg integration for HeliosDB. Enables OLTP+OLAP on open table formats with snapshot isolation, schema evolution, and partition management.

Iceberg multi-catalog support (Hive, Glue, Nessie, REST) is in the main lakehouse README’s feature list. Schema evolution and partition evolution work as you’d expect — schema changes are recorded in the metadata log, queries against historical snapshots see the schema as of that snapshot.

6. Delta-Specific Operations

From heliosdb-lakehouse/crates/delta:

use heliosdb_lakehouse_delta::{DeltaLakeManager, DeltaTableConfig};
use arrow::datatypes::{DataType, Field, Schema};
use std::sync::Arc;


let mut config = DeltaTableConfig::default();
config.table_uri = "s3://my-bucket/delta-tables".to_string();
config.enable_z_ordering = true;
config.enable_time_travel = true;


let manager = DeltaLakeManager::new(config).await?;


let schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int32, false),
    Field::new("name", DataType::Utf8, false),
    Field::new("age", DataType::Int32, true),
]));


manager.create_table("users", schema).await?;

Delta features supported per the README:

  • ACID transactions, full CRUD
  • Time travel by version or timestamp
  • Z-ordering for multi-dimensional clustering
  • Data skipping (statistics-based file pruning)
  • Databricks Unity Catalog and Delta Sharing protocol

Storage backends: S3, Azure Blob, GCS, local.

7. Hudi Operations (Active, 20%)

From the main lakehouse README:

use heliosdb_lakehouse::*;


let config = HudiConfig {
    base_path: "s3://bucket/hudi/table".to_string(),
    table_name: "orders".to_string(),
    table_type: "MERGE_ON_READ".to_string(),
    ..Default::default()
};


let adapter = HudiAdapter::new(config).await?;


// Incremental query — fetch only changes between two commit times
let batches = adapter.incremental_query(
    "s3://bucket/hudi/table",
    "20240101120000",
    Some("20240101130000")
).await?;


// Compaction
let compaction_config = CompactionConfig {
    strategy: CompactionStrategy::Async,
    target_file_size: 128 * 1024 * 1024,
    ..Default::default()
};
adapter.compact("s3://bucket/hudi/table", compaction_config).await?;

Hudi’s headline feature is incremental queries — give it a from/to commit timeline and you get only changed rows. Useful for CDC pipelines and incremental ETL.

8. Performance Targets

Per the unified-API benchmarks in the README:

OperationLatency
Table registration<1ms
Format detection<0.1ms
SQL parsing<0.5ms
Catalog operations<10ms

These are catalog ops — actual query latency is dominated by scan cost from object storage. The format adapters push filters and projections down so you only fetch what the query needs.

9. Catalog Registry

When you have many tables, register them all up front and let the catalog drive:

use heliosdb_lakehouse::*;


let manager = LakehouseManager::new().await?;
let catalog = manager.catalog();


catalog.register_namespace("analytics").await?;
catalog.register_table("analytics.orders",   "s3://bucket/orders").await?;
catalog.register_table("analytics.customers", "gs://bucket/customers").await?;
catalog.register_table("analytics.products",  "az://bucket/products").await?;

Now analytics.orders is just another schema-qualified relational name.

10. Honest Status (From the README)

  • Iceberg: 25% complete per README (table metadata, snapshots, schema/partition evolution, time travel, multi-catalog).
  • Delta: 25% complete per README (transaction log, version history, time travel, schema merging, OPTIMIZE/VACUUM, Z-ordering, Unity Catalog).
  • Hudi: 20% complete per README (COW + MOR tables, incremental queries, compaction, timeline).
  • Unified API: 10% complete (single API, format detection, SQL interface, unified time travel, cross-format federation).

Test coverage: 70+ integration tests covering registration, format detection, SQL parsing, all three adapters, time travel, multi-format scenarios.

11. Future Enhancements (From README)

  • Cross-format time travel queries (currently per-format).
  • Unified write operations (INSERT, UPDATE, DELETE) across formats.
  • Catalog auto-sync.
  • Caching and parallel-operation optimizations.

Today, write operations are best done through each format’s native adapter directly.

Where Next