RAG Pipelines with L3 Semantic Caching
Semantic caching for RAG pipelines on HeliosDB Lite, where a question whose embedding is close to an earlier one is answered from the cached retrieval instead of a new vector search, is planned in HeliosProxy. Today the proxy's opt-in query cache serves repeated read queries from its in-process L1 and L2 tiers and refuses an entry once a write through the proxy commits to a table it read.
Planned: semantic (similarity-matched) result caching in HeliosProxy is on the roadmap. Read the documentation.
Concrete Examples with Code, Config & Architecture
Example 1: Embedded Configuration for L3 Semantic Cache
Configuration: helios_semantic_cache.toml
[helios]
data_dir = "/var/lib/helios-data"
mode = "server"
[semantic_cache]
# Enable L3 semantic caching for vector similarity queries
enabled = true
# Cache storage backend
storage = "memory" # "memory" | "disk" | "hybrid"
memory_limit = "4GB"
disk_path = "/var/lib/helios-cache"
[semantic_cache.embedding]
# Embedding configuration
embedding_dimension = 768 # Match your embedding model (OpenAI: 1536, Anthropic: 768)
projection_dimension = 128 # Reduced dimension for faster hashing
# Embedding similarity metric
similarity_metric = "cosine" # "cosine" | "euclidean" | "dot_product"
[semantic_cache.lsh]
# Locality-Sensitive Hashing configuration
num_hash_functions = 3 # More functions = higher recall, more memory
hash_function = "simhash" # "simhash" | "minhash"
random_seed = 42 # Reproducible hash generation
# Similarity threshold for cache hits
similarity_threshold = 0.95 # 0.90-0.99; higher = stricter matching
auto_tune_threshold = true # Automatically adjust based on hit rate
[semantic_cache.eviction]
# Cache eviction policy
policy = "adaptive" # "lru" | "lfu" | "adaptive" (RL-based)
max_entries = 100000
ttl_default = "24h"
ttl_min = "1h"
ttl_max = "48h"
# Adaptive eviction (reinforcement learning)
rl_optimization = true
rl_reward_function = "cost_savings" # Optimize for cost reduction
rl_update_interval = "5m"
[semantic_cache.warming]
# Cache warming strategies
auto_warm = true
warm_on_miss = true # Asynchronously warm cache after misses
warm_batch_size = 100
warm_interval = "10m"
# Pre-warming for common queries
prewarm_enabled = true
prewarm_query_log = "/var/log/helios/query.log"
prewarm_threshold = 5 # Warm queries seen ≥5 times
[semantic_cache.verification]
# Cache accuracy verification
enabled = true
spot_check_rate = 0.01 # Verify 1% of cache hits
max_drift_tolerance = 0.02 # Maximum allowed similarity drift
invalidate_on_drift = true
[semantic_cache.observability]
# Metrics and monitoring
metrics_enabled = true
metrics_port = 9092
# Prometheus metrics:
# - semantic_cache_hit_rate
# - semantic_cache_lookup_duration_seconds
# - semantic_cache_memory_usage_bytes
# - semantic_cache_cost_savings_total
# - semantic_cache_similarity_threshold (current adaptive value)
log_cache_hits = false # Verbose logging (debugging only)
log_cache_misses = false
[pgvector]
# pgvector integration
enabled = true
intercept_similarity_queries = true
min_limit_for_caching = 1 # Cache queries with LIMIT ≥ 1
max_limit_for_caching = 100 # Don't cache very large result sets
# Query patterns to cache
cache_query_patterns = [
"embedding <=> $1", # Cosine distance
"embedding <-> $1", # Euclidean distance
"embedding <#> $1", # Negative dot product
]
Example 2: Language Binding Integration (Python)
(Continuing in next message due to length...)
Performance depends on workload and hardware. Published figures are measured on representative fixtures; see the benchmarks page and reproduce them on your own workload.