---
title: "Complex Event Processing (CEP) Patterns Guide"
---

# Complex Event Processing (CEP) Patterns Guide

:::caution[Planned: streaming engine]
The streaming engine for HeliosDB Full is planned. The current release runs no stream
processing, continuous queries or windowed stream joins, and the SQL on this page is not
accepted yet; this page describes the planned design. To see recent changes today, run
`SHOW CDC EVENTS [LIMIT n]` (see the [CDC tutorial](/docs/full/tutorials/intermediate/05-cdc-events/)).
:::

## Overview

Complex Event Processing enables pattern detection in event streams using MATCH_RECOGNIZE SQL syntax and NFA (Non-deterministic Finite Automaton) matching. The planned HeliosDB Streaming engine provides CEP capabilities for detecting complex patterns across time-ordered events.

## Pattern Types

### 1. Sequence Patterns

Detect ordered sequences of events.

**Use Case**: User journey tracking (view → add to cart → purchase)

**SQL Example**:
```sql
SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        FIRST(A.event_time) AS journey_start,
        LAST(C.event_time) AS journey_end,
        (C.event_time - A.event_time) AS conversion_time
    PATTERN (A B C)
    DEFINE
        A AS A.event_type = 'page_view',
        B AS B.event_type = 'add_to_cart',
        C AS C.event_type = 'purchase'
)
```

### 2. Missing Event Patterns

Detect absence of expected events within a time window.

**Use Case**: Abandoned cart detection

**SQL Example**:
```sql
SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        A.cart_id AS abandoned_cart,
        A.event_time AS cart_time,
        CURRENT_TIMESTAMP AS detected_at
    PATTERN (A B* NOT C)
    WITHIN INTERVAL '1' HOUR
    DEFINE
        A AS A.event_type = 'add_to_cart',
        B AS B.event_type != 'purchase' AND B.event_type != 'cart_clear',
        C AS C.event_type = 'purchase'
)
```

### 3. Looping Patterns

Detect repeated occurrences of events.

**Use Case**: Failed login detection (3+ failed attempts)

**SQL Example**:
```sql
SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        COUNT(A.*) AS failed_attempts,
        FIRST(A.event_time) AS first_failure,
        LAST(A.event_time) AS last_failure
    PATTERN (A{3,})
    WITHIN INTERVAL '5' MINUTE
    DEFINE
        A AS A.event_type = 'login_failed'
)
```

### 4. Alternation Patterns

Multiple possible event sequences.

**Use Case**: Payment method selection

**SQL Example**:
```sql
SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
        CASE
            WHEN MATCH_NUMBER(B) = 1 THEN 'credit_card'
            WHEN MATCH_NUMBER(C) = 1 THEN 'paypal'
            ELSE 'unknown'
        END AS payment_method
    PATTERN ((A B) | (A C))
    DEFINE
        A AS A.event_type = 'checkout_started',
        B AS B.event_type = 'credit_card_selected',
        C AS C.event_type = 'paypal_selected'
)
```

### 5. Conditional Patterns

Patterns with conditional logic.

**Use Case**: High-value transaction anomaly detection

**SQL Example**:
```sql
SELECT *
FROM events
MATCH_RECOGNIZE (
    PARTITION BY account_id
    ORDER BY event_time
    MEASURES
        A.amount AS suspicious_amount,
        AVG(B.amount) AS avg_normal_amount,
        (A.amount / AVG(B.amount)) AS anomaly_ratio
    PATTERN (B{5,} A)
    WITHIN INTERVAL '1' DAY
    DEFINE
        B AS B.amount BETWEEN 10 AND 100,
        A AS A.amount > AVG(B.amount) * 5
)
```

## Performance Considerations

### 1. NFA State Explosion

Complex patterns can create large state machines. Mitigation:
- Limit pattern complexity (max 5-7 operators per pattern)
- Use time constraints to bound state lifetime
- Partition by key to distribute load

### 2. Memory Usage

Each active pattern instance maintains state.

### 3. Latency Optimization

- **Early Termination**: Use `NOT` patterns for fast rejection
- **Pattern Ordering**: Most selective conditions first
- **Partition Pruning**: Partition by high-cardinality keys

## Troubleshooting

### High Memory Usage

**Symptom**: OOM errors or high GC pressure

**Solution**:
1. Reduce `max_pattern_instances`
2. Decrease time windows
3. Add more aggressive state cleanup

### Slow Pattern Matching

**Symptom**: High latency in pattern matching

**Solution**:
1. Simplify pattern complexity
2. Increase parallelism
3. Optimize event predicates (avoid expensive operations)

### Missing Matches

**Symptom**: Expected patterns not detected

**Solution**:
1. Check event ordering (ORDER BY in SQL)
2. Verify partition keys
3. Increase time windows
4. Enable debug logging

## References

- [Flink CEP Documentation](https://nightlies.apache.org/flink/flink-docs-stable/docs/libs/cep/)
- [SQL MATCH_RECOGNIZE Syntax](https://www.iso.org/standard/76584.html)

---

*Version: 1.0*