--- title: "Time-Series Features in HeliosDB" --- # Time-Series Features in HeliosDB --- ## Overview HeliosDB provides time-series data management for IoT, observability, financial analytics, and log processing workloads. The time-series engine delivers sub-millisecond query latency, 10x+ compression ratios, and throughput exceeding 1M+ points per second. ## Key Capabilities | Feature | Description | Performance | |---------|-------------|-------------| | **Native Time-Series Storage** | Columnar storage optimized for time-ordered data | Zero-copy batch operations | | **Gorilla Compression** | Facebook's industry-standard compression algorithm | 10-15x compression ratio | | **Time-Based Partitioning** | Hourly/daily/weekly/monthly/yearly partitions | Partition pruning for queries | | **Retention Policies** | Automatic data expiration with TTL and size limits | Background cleanup | | **Downsampling** | Multi-tier aggregation with configurable intervals | Preserves statistical properties | | **Continuous Aggregates** | Pre-computed rollups for fast analytics | Real-time materialization | | **Window Functions** | Tumbling, sliding, and session windows | Time-based analysis | | **Gap Filling** | Interpolation strategies for missing data | Forward/backward/linear fill | --- ## Architecture ``` HeliosDB Time-Series Architecture ┌──────────────────────────────────────────────────────────────────────┐ │ Time-Series API Layer │ │ write_point() | query_range() | downsample() | set_retention() │ ├──────────────────────────────────────────────────────────────────────┤ │ High-Performance Ingestion │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │ │ Batching │ │Out-of-Order│ │ Backfill │ │ Compression │ │ │ │ Buffer │ │ Handler │ │ Support │ │ Pipeline │ │ │ └────────────┘ └────────────┘ └────────────┘ └────────────────┘ │ ├──────────────────────────────────────────────────────────────────────┤ │ Query Engine Layer │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │ │Time Range │ │ Window │ │ Time-Based │ │ Result │ │ │ │ Queries │ │ Functions │ │ Joins │ │ Caching │ │ │ └────────────┘ └────────────┘ └────────────┘ └────────────────┘ │ ├──────────────────────────────────────────────────────────────────────┤ │ Data Management Layer │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │ │ Retention │ │Downsampling│ │ Partition │ │ Tiered │ │ │ │ Engine │ │ Engine │ │ Manager │ │ Storage │ │ │ └────────────┘ └────────────┘ └────────────┘ └────────────────┘ │ ├──────────────────────────────────────────────────────────────────────┤ │ Compression Layer │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Gorilla Compressor │ │ │ │ Delta-of-Delta (Timestamps) | XOR Bitpacking (Values) │ │ │ │ Dictionary Compression (Metrics/Tags) │ │ │ └─────────────────────────────────────────────────────────────────┘ │ ├──────────────────────────────────────────────────────────────────────┤ │ LSM Storage Engine │ └──────────────────────────────────────────────────────────────────────┘ ``` --- ## Native Time-Series Storage Engine HeliosDB's time-series storage is built on a columnar architecture that separates timestamps, values, and metadata for optimal compression and query performance. --- ## Time-Bucketed Aggregations HeliosDB supports automatic time-bucketing for analytical queries with multiple aggregation functions. ### Aggregation Functions | Function | Description | |----------|-------------| | `Average` | Mean of all values in bucket | | `Min` | Minimum value | | `Max` | Maximum value | | `Sum` | Sum of all values | | `Count` | Number of data points | | `First` | First value in bucket | | `Last` | Last value in bucket | | `StdDev` | Standard deviation | | `Percentile(n)` | Nth percentile | ### Example: Time Bucket Query ```sql -- Aggregate sensor readings by 5-minute buckets SELECT time_bucket('5 minutes', timestamp) AS bucket, sensor_id, AVG(temperature) AS avg_temp, MAX(temperature) AS max_temp, MIN(temperature) AS min_temp FROM sensor_readings WHERE timestamp BETWEEN '2025-01-01' AND '2025-01-02' GROUP BY bucket, sensor_id ORDER BY bucket; ``` --- ## Downsampling and Retention Policies ### Multi-Tier Downsampling Configure cascading downsampling tiers to reduce storage while preserving analytical value. --- ## Continuous Aggregates Pre-compute rollups for commonly accessed time ranges. ### Benefits - **Query Performance**: Pre-computed results eliminate runtime aggregation - **Storage Efficiency**: Aggregated data is smaller than raw data - **Real-Time Updates**: Continuous background processing - **Late Data Handling**: Configurable lag for out-of-order points --- ## Time-Based Partitioning ### Partition Strategies | Strategy | Partition ID Format | Use Case | |----------|-------------------|----------| | `Hourly` | YYYYMMDDHH | High-frequency data | | `Daily` | YYYYMMDD | Standard metrics | | `Weekly` | YYYYWW | Low-frequency data | | `Monthly` | YYYYMM | Long-term storage | | `Yearly` | YYYY | Historical archives | | `Custom(secs)` | timestamp/interval | Flexible partitioning | --- ## Compression Performance ### Gorilla Algorithm Results | Data Type | Compression Ratio | Throughput | |-----------|-------------------|------------| | IoT Temperature | 8-12x | 500K+ pts/sec | | Network Metrics | 5-8x | 500K+ pts/sec | | CPU/System Metrics | 5-10x | 500K+ pts/sec | | High-Frequency Trading | 10-15x | 500K+ pts/sec | ### Compression Pipeline 1. **Delta-of-Delta Encoding** (Timestamps) - Regular intervals compress to 1-4 bits per timestamp - 16-64x compression for uniformly sampled data 2. **XOR + Bit-packing** (Values) - Exploits temporal correlation in values - 4-20x compression for slowly changing values 3. **Dictionary Compression** (Metrics/Tags) - String to u32 ID mapping - 10-20x reduction for metric names --- ## Window Functions ### Window Types - **Tumbling**: Fixed-size, non-overlapping windows - **Sliding**: Fixed-size, overlapping windows - **Session**: Dynamic windows based on inactivity gap --- ## Gap Filling and Interpolation ### Fill Strategies | Strategy | Description | |----------|-------------| | `Null` | Leave gaps as null/None | | `Zero` | Fill with zero values | | `Forward` | Use previous known value | | `Backward` | Use next known value | | `Linear` | Linear interpolation between points | --- ## Time-Zone Handling HeliosDB stores all timestamps in UTC and provides time-zone conversion at query time: ```sql -- Query with timezone conversion SELECT timestamp AT TIME ZONE 'America/New_York' AS local_time, value FROM metrics WHERE timestamp > NOW() - INTERVAL '24 hours'; ``` --- ## Integration with MVCC Time-series data integrates with HeliosDB's Multi-Version Concurrency Control: - **Point-in-Time Queries**: Query data as it existed at any past moment - **Consistent Snapshots**: Transactional reads across time ranges - **Conflict-Free Writes**: Append-only model eliminates write conflicts --- ## Related Documentation | Document | Description | |----------|-------------| | [Quick Start Guide](https://www.heliosdb.com/docs/full/features/time-series/quick_start/) | Get started in 10 minutes | | [Examples](https://www.heliosdb.com/docs/full/features/time-series/examples/) | Code examples for common use cases | --- ## Performance Targets Indicative figures, measured on representative fixtures; reproduce on your own hardware. | Metric | Target | Achieved | |--------|--------|----------| | Ingestion throughput | 1M pts/sec | 500K+ pts/sec | | Compression ratio | 8-10x | 10-15x | | Compression latency | <5ms/1K pts | <3ms/1K pts | | Decompression latency | <3ms/1K pts | <2ms/1K pts | | Query latency (time range) | <10ms | <5ms | | Partition pruning | 95% reduction | 95%+ | --- ## Use Cases ### IoT and Sensor Networks - High-volume sensor data ingestion - Edge device telemetry - Industrial monitoring ### Observability and Monitoring - Infrastructure metrics (CPU, memory, disk) - Application performance monitoring - Log aggregation and analysis ### Financial Data - High-frequency trading ticks - Market data feeds - Risk analytics ### Operational Analytics - Real-time dashboards - Anomaly detection - Trend analysis --- **See Also**: HeliosDB Feature Index