Feature 08: Multi-Tenancy Support

Feature 08: Multi-Tenancy Support

Priority: High | Complexity: High | Phase: 3 (Enterprise)

---

Overview

Problem Statement

SaaS applications need to serve multiple customers (tenants) from shared infrastructure:

• Each tenant’s data must be isolated
• Resource usage must be fair across tenants
• Costs must be attributable per tenant
• Tenant-specific policies (rate limits, quotas) needed

Without multi-tenancy at proxy level:

• Each tenant needs separate database instances (expensive)
• Or complex application-level tenant isolation (error-prone)
• No cross-tenant visibility for platform operators

Solution

Implement multi-tenancy at the proxy layer with multiple isolation strategies:

                    ┌─────────────────────────────────────────────────┐
                    │              MULTI-TENANT PROXY                  │
                    │                                                  │
  Tenant A ────────►│  ┌──────────────────────────────────────────┐   │
                    │  │ Tenant Identification                    │   │
  Tenant B ────────►│  │ - Header (X-Tenant-Id)                   │   │
                    │  │ - Username prefix (tenant_a.user)        │   │
  Tenant C ────────►│  │ - Database name (tenant_a_db)            │   │
                    │  │ - JWT claim                              │   │
                    │  └──────────────────────────────────────────┘   │
                    │                    │                             │
                    │                    ▼                             │
                    │  ┌──────────────────────────────────────────┐   │
                    │  │ Isolation Strategy                       │   │
                    │  │                                          │   │
                    │  │  ┌─────────┐ ┌─────────┐ ┌─────────┐    │   │
                    │  │  │Database │ │ Schema  │ │  Row    │    │   │
                    │  │  │per-Tenant│ │per-Tenant│ │per-Tenant│   │   │
                    │  │  └─────────┘ └─────────┘ └─────────┘    │   │
                    │  └──────────────────────────────────────────┘   │
                    │                    │                             │
                    │                    ▼                             │
                    │  ┌──────────────────────────────────────────┐   │
                    │  │ Tenant Policies                          │   │
                    │  │ - Rate limits                            │   │
                    │  │ - Connection pools                       │   │
                    │  │ - Query restrictions                     │   │
                    │  └──────────────────────────────────────────┘   │
                    └─────────────────────────────────────────────────┘

---

Architecture

Tenant Manager

pub struct TenantManager {
    /// Tenant configurations
    tenants: DashMap<TenantId, TenantConfig>,

    /// Tenant identification strategy
    identifier: Box<dyn TenantIdentifier>,

    /// Default tenant config
    default_config: TenantConfig,
}

#[derive(Debug, Clone)]
pub struct TenantConfig {
    /// Tenant identifier
    pub id: TenantId,

    /// Display name
    pub name: String,

    /// Isolation strategy
    pub isolation: IsolationStrategy,

    /// Rate limits
    pub rate_limits: TenantRateLimits,

    /// Connection pool settings
    pub pool: TenantPoolConfig,

    /// Allowed operations
    pub permissions: TenantPermissions,

    /// Custom metadata
    pub metadata: HashMap<String, String>,
}

#[derive(Debug, Clone)]
pub enum IsolationStrategy {
    /// Separate database per tenant
    Database {
        database_name: String,
    },

    /// Separate schema per tenant (same database)
    Schema {
        database_name: String,
        schema_name: String,
    },

    /// Row-level security (same tables)
    Row {
        database_name: String,
        tenant_column: String,
    },

    /// Branch per tenant (HeliosDB-Lite specific)
    Branch {
        branch_name: String,
    },
}

pub trait TenantIdentifier: Send + Sync {
    fn identify(&self, request: &Request) -> Option<TenantId>;
}

Tenant Identification Strategies

pub struct HeaderTenantIdentifier {
    header_name: String,
}

impl TenantIdentifier for HeaderTenantIdentifier {
    fn identify(&self, request: &Request) -> Option<TenantId> {
        request.headers()
            .get(&self.header_name)
            .map(|v| TenantId(v.to_string()))
    }
}

pub struct UsernamePrefixIdentifier {
    separator: char,
}

impl TenantIdentifier for UsernamePrefixIdentifier {
    fn identify(&self, request: &Request) -> Option<TenantId> {
        // Username format: tenant_id.actual_username
        request.username()
            .split(self.separator)
            .next()
            .map(|t| TenantId(t.to_string()))
    }
}

pub struct JwtClaimIdentifier {
    claim_name: String,
    jwt_verifier: Arc<JwtVerifier>,
}

impl TenantIdentifier for JwtClaimIdentifier {
    fn identify(&self, request: &Request) -> Option<TenantId> {
        let token = request.auth_token()?;
        let claims = self.jwt_verifier.verify(token).ok()?;
        claims.get(&self.claim_name)
            .and_then(|v| v.as_str())
            .map(|t| TenantId(t.to_string()))
    }
}

Tenant-Aware Connection Pool

pub struct TenantConnectionPool {
    /// Per-tenant connection pools
    pools: DashMap<TenantId, ConnectionPool>,

    /// Shared pool for small tenants
    shared_pool: ConnectionPool,

    /// Configuration
    config: TenantPoolManagerConfig,
}

impl TenantConnectionPool {
    pub fn acquire(&self, tenant: &TenantId) -> Connection {
        // Large tenants get dedicated pools
        if let Some(pool) = self.pools.get(tenant) {
            return pool.acquire();
        }

        // Small tenants share pool
        self.shared_pool.acquire()
    }

    pub fn get_pool_for_tenant(&self, tenant: &TenantId, config: &TenantConfig) -> ConnectionPool {
        self.pools.entry(tenant.clone())
            .or_insert_with(|| {
                ConnectionPool::new(ConnectionPoolConfig {
                    max_size: config.pool.max_connections,
                    min_idle: config.pool.min_idle,
                    ..Default::default()
                })
            })
            .clone()
    }
}

Query Transformation

pub struct TenantQueryTransformer;

impl TenantQueryTransformer {
    /// Transform query for row-level isolation
    pub fn inject_tenant_filter(
        &self,
        query: &str,
        tenant: &TenantId,
        config: &TenantConfig,
    ) -> String {
        if let IsolationStrategy::Row { tenant_column, .. } = &config.isolation {
            // Parse and inject WHERE clause
            let mut ast = self.parse(query);
            self.add_tenant_filter(&mut ast, tenant_column, &tenant.0);
            return self.to_sql(&ast);
        }

        query.to_string()
    }

    /// Set search_path for schema isolation
    pub fn set_schema_search_path(&self, tenant: &TenantId, config: &TenantConfig) -> Option<String> {
        if let IsolationStrategy::Schema { schema_name, .. } = &config.isolation {
            return Some(format!("SET search_path TO {}", schema_name));
        }
        None
    }
}

---

API Specification

Configuration (heliosproxy.toml)

[multi_tenancy]
enabled = true

# How to identify tenants
identification = "header"    # header, username_prefix, jwt, database

[multi_tenancy.identification.header]
name = "X-Tenant-Id"

[multi_tenancy.identification.jwt]
claim = "tenant_id"
issuer = "https://auth.example.com"

# Default tenant config
[multi_tenancy.default]
isolation = "schema"
max_connections = 10
qps_limit = 100

# Per-tenant overrides
[multi_tenancy.tenants.tenant_a]
name = "Acme Corp"
isolation = "database"
database = "tenant_a_db"
max_connections = 50
qps_limit = 1000

[multi_tenancy.tenants.tenant_b]
name = "Beta Inc"
isolation = "schema"
database = "shared_db"
schema = "tenant_b"
max_connections = 20
qps_limit = 500

[multi_tenancy.tenants.tenant_c]
name = "Gamma LLC"
isolation = "row"
database = "shared_db"
tenant_column = "tenant_id"
max_connections = 10
qps_limit = 200

# HeliosDB-Lite specific: branch isolation
[multi_tenancy.tenants.tenant_d]
name = "Delta Labs"
isolation = "branch"
branch = "tenant_d"
max_connections = 30

Admin API

GET /tenants
{
  "tenants": [
    {
      "id": "tenant_a",
      "name": "Acme Corp",
      "isolation": "database",
      "active_connections": 25,
      "qps_current": 456,
      "qps_limit": 1000
    }
  ]
}

POST /tenants
# Create new tenant
{
  "id": "tenant_e",
  "name": "Echo Inc",
  "isolation": "schema",
  "database": "shared_db",
  "schema": "tenant_e",
  "max_connections": 20,
  "qps_limit": 300
}

GET /tenants/{tenant_id}/stats
{
  "tenant_id": "tenant_a",
  "connections": {
    "active": 25,
    "idle": 10,
    "max": 50
  },
  "queries": {
    "total": 150000,
    "qps_current": 456,
    "qps_limit": 1000,
    "slow_queries": 12
  },
  "data": {
    "rows_read": 5000000,
    "rows_written": 50000,
    "bytes_read": "1.2GB",
    "bytes_written": "50MB"
  }
}

PUT /tenants/{tenant_id}/limits
# Update tenant limits
{
  "max_connections": 100,
  "qps_limit": 2000
}

DELETE /tenants/{tenant_id}
# Disable tenant access

Tenant Context SQL

-- Application sets tenant context
SET helios.tenant_id = 'tenant_a';

-- Queries automatically filtered
SELECT * FROM users;  -- Filtered to tenant_a's data

-- Explicit tenant override (admin only)
/*helios:tenant=tenant_b*/ SELECT COUNT(*) FROM users;

---

AI/Agent Innovations

1. Per-Tenant AI Workload Isolation

Isolate AI workloads per tenant:

pub struct TenantAiIsolation {
    /// Per-tenant embedding caches
    embedding_caches: DashMap<TenantId, EmbeddingCache>,

    /// Per-tenant vector indexes
    vector_indexes: DashMap<TenantId, VectorIndex>,

    /// Per-tenant token budgets
    token_budgets: DashMap<TenantId, TokenBudget>,
}

impl TenantAiIsolation {
    pub fn get_embedding_cache(&self, tenant: &TenantId) -> &EmbeddingCache {
        self.embedding_caches.entry(tenant.clone())
            .or_insert_with(EmbeddingCache::new)
    }

    pub fn consume_tokens(&self, tenant: &TenantId, tokens: u64) -> Result<(), QuotaExceeded> {
        let budget = self.token_budgets.get(tenant)?;
        budget.consume(tokens)
    }
}

2. Tenant-Specific RAG Contexts

Isolate knowledge bases per tenant:

[multi_tenancy.tenants.tenant_a.ai]
knowledge_base = "tenant_a_docs"
embedding_model = "custom-fine-tuned"
retrieval_limit = 20

[multi_tenancy.tenants.tenant_b.ai]
knowledge_base = "tenant_b_docs"
embedding_model = "default"
retrieval_limit = 10

3. Agent Workspace Isolation

Each tenant gets isolated agent workspaces:

pub struct TenantAgentWorkspace {
    /// Conversation history per tenant
    conversations: DashMap<TenantId, ConversationStore>,

    /// Tool permissions per tenant
    tool_permissions: DashMap<TenantId, ToolPermissions>,

    /// Agent quotas per tenant
    quotas: DashMap<TenantId, AgentQuota>,
}

4. Cross-Tenant Analytics (Platform View)

Platform operators see aggregated analytics:

GET /analytics/cross-tenant
{
  "summary": {
    "total_tenants": 150,
    "active_tenants": 87,
    "total_qps": 45000
  },
  "top_tenants_by_usage": [
    {"tenant": "tenant_a", "qps": 5000, "connections": 45}
  ],
  "resource_distribution": {
    "connections_used_pct": 0.65,
    "qps_used_pct": 0.78
  }
}

---

HeliosDB-Lite Integration

1. Branch-Based Tenant Isolation

Use HeliosDB-Lite branches for perfect isolation:

pub struct BranchTenantIsolation {
    /// Tenant to branch mapping
    tenant_branches: HashMap<TenantId, BranchName>,
}

impl BranchTenantIsolation {
    pub fn setup_tenant(&self, tenant: &TenantId) -> Result<()> {
        // Create branch for tenant
        let branch_name = format!("tenant_{}", tenant.0);
        self.db.create_branch(&branch_name, "main")?;

        self.tenant_branches.insert(tenant.clone(), branch_name);
        Ok(())
    }

    pub fn route_query(&self, tenant: &TenantId, query: &str) -> RoutingDecision {
        if let Some(branch) = self.tenant_branches.get(tenant) {
            // Route to tenant's branch
            return RoutingDecision::branch(branch);
        }

        RoutingDecision::default()
    }
}

2. Per-Tenant Time Travel

Each tenant gets isolated history:

-- Tenant A's history
SET helios.tenant_id = 'tenant_a';
SELECT * FROM orders AS OF '2025-12-01T00:00:00Z';

-- Tenant B cannot see tenant A's history
SET helios.tenant_id = 'tenant_b';
SELECT * FROM orders AS OF '2025-12-01T00:00:00Z';  -- Only tenant_b data

3. Tenant Data Export

Export tenant data with branch snapshots:

pub async fn export_tenant_data(&self, tenant: &TenantId) -> Result<Export> {
    let branch = self.get_tenant_branch(tenant)?;

    // Create snapshot of tenant's branch
    let snapshot = self.db.create_snapshot(branch)?;

    // Export as portable format
    self.db.export_branch(&snapshot)
}

4. Tenant Migration

Migrate tenant between isolation strategies:

pub async fn migrate_tenant(
    &self,
    tenant: &TenantId,
    from: IsolationStrategy,
    to: IsolationStrategy,
) -> Result<()> {
    match (&from, &to) {
        // Schema to Branch migration
        (IsolationStrategy::Schema { .. }, IsolationStrategy::Branch { branch_name }) => {
            // Create branch
            self.db.create_branch(branch_name, "main")?;

            // Copy data to branch
            self.copy_schema_to_branch(tenant, branch_name).await?;

            // Update tenant config
            self.update_tenant_isolation(tenant, to)?;
        }
        _ => {
            return Err(Error::UnsupportedMigration);
        }
    }

    Ok(())
}

---

Implementation Notes

File Locations

src/proxy/
├── multi_tenancy/
│   ├── mod.rs              # Public API
│   ├── manager.rs          # TenantManager
│   ├── identifier.rs       # TenantIdentifier implementations
│   ├── isolation.rs        # IsolationStrategy implementations
│   ├── pool.rs             # TenantConnectionPool
│   ├── transformer.rs      # TenantQueryTransformer
│   └── metrics.rs          # Per-tenant metrics

Key Considerations

1. Cross-Tenant Queries: Admin users may need cross-tenant access. Implement carefully with audit logging.

2. Tenant Provisioning: Automate tenant setup (create database/schema/branch).

3. Tenant Deletion: Handle data cleanup carefully, consider soft-delete with retention.

4. Noisy Neighbors: Implement fair scheduling to prevent one tenant from impacting others.

5. Compliance: Consider data residency requirements per tenant.

Security Checklist

• Tenant identification cannot be spoofed
• Row-level filters cannot be bypassed
• Cross-tenant queries require explicit admin privilege
• Audit log captures tenant context
• Tenant credentials are isolated

Performance Targets

Metric	Target	Measurement
Tenant identification	<10μs	p99
Query transformation	<100μs	p99 (row-level)
Pool isolation overhead	<5%	compared to single-tenant

---

Related Features

• Rate Limiting - Per-tenant limits
• Query Analytics - Per-tenant analytics
• Authentication Proxy - Tenant auth