Tool Result Caching for AI Agents

Problem Being Solved

Core Problem Statement

AI agents repeatedly execute identical or semantically similar tool calls across sessions and users, generating unnecessary API costs, increased latency, and degraded user experience. Current solutions require complex distributed caching infrastructure that introduces operational overhead, fails to understand semantic equivalence, and lacks fine-grained invalidation strategies for dynamic tool results.

Concrete Examples with Code, Config & Architecture

Example 1: Embedded Configuration for AI Agent with Tool Caching

TOML Configuration (helios_agent_cache.toml):

[database]
type = "embedded"
path = "./agent_cache.db"
mode = "readwrite-create"
page_size = 4096
cache_size_mb = 512
wal_mode = true
busy_timeout_ms = 5000

[helios_proxy]
enabled = true
semantic_matching = true
embedding_model = "all-MiniLM-L6-v2"
similarity_threshold = 0.92

[cache_policies]
# Default TTL for unknown tools
default_ttl_seconds = 3600

# Per-tool TTL overrides
[cache_policies.tools]
"weather.current" = 300        # 5 minutes
"stock.price" = 60             # 1 minute (volatile)
"company.info" = 86400         # 24 hours (stable)
"geocoding.address" = 604800   # 7 days (very stable)
"calculator.*" = 31536000      # 1 year (deterministic)

[invalidation_rules]
enabled = true

# When location changes, invalidate all weather calls for that location
[[invalidation_rules.triggers]]
source_table = "user_locations"
source_event = "UPDATE"
target_cache_pattern = "weather.*"
match_field = "location_id"

[performance]
cache_hit_target = 0.85
max_embedding_batch_size = 32
vector_index_type = "hnsw"  # Hierarchical Navigable Small World
hnsw_ef_construction = 200
hnsw_m = 16

[monitoring]
log_cache_hits = true
log_slow_queries_ms = 100
export_metrics_prometheus = true
metrics_port = 9091

Example 3: Infrastructure & Container Deployment

Dockerfile for AI Agent with Embedded Cache:

FROM rust:1.75-slim as builder

WORKDIR /app

# Install dependencies
RUN apt-get update && apt-get install -y \
    pkg-config \
    libssl-dev \
    && rm -rf /var/lib/apt/lists/*

# Copy project files
COPY Cargo.toml Cargo.lock ./
COPY src ./src

# Build release binary
RUN cargo build --release --bin ai-agent-service

# Runtime stage
FROM debian:bookworm-slim

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    ca-certificates \
    libssl3 \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Copy binary from builder
COPY --from=builder /app/target/release/ai-agent-service /usr/local/bin/

# Copy configuration
COPY config/helios_agent_cache.toml /app/config/

# Create directory for database files
RUN mkdir -p /app/data && chmod 777 /app/data

# Environment variables
ENV HELIOS_DB_PATH=/app/data/agent_cache.db
ENV HELIOS_CONFIG=/app/config/helios_agent_cache.toml
ENV RUST_LOG=info

EXPOSE 8080

CMD ["ai-agent-service"]

Docker Compose for Multi-Agent Deployment:

version: '3.8'

services:
  ai-agent-api:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: ai-agent-api
    ports:
      - "8080:8080"
      - "9091:9091"  # Prometheus metrics
    volumes:
      - agent-cache-data:/app/data
      - ./config:/app/config:ro
    environment:
      - HELIOS_DB_PATH=/app/data/agent_cache.db
      - HELIOS_CONFIG=/app/config/helios_agent_cache.toml
      - AGENT_POOL_SIZE=10
      - MAX_CACHE_SIZE_MB=1024
      - CACHE_HIT_TARGET=0.85
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '1'
          memory: 512M

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
    depends_on:
      - ai-agent-api
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./grafana/datasources:/etc/grafana/provisioning/datasources:ro
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  agent-cache-data:
    driver: local
  prometheus-data:
    driver: local
  grafana-data:
    driver: local

Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-agent-service
  namespace: ai-platform
  labels:
    app: ai-agent
    version: v1.0.0
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-agent
  template:
    metadata:
      labels:
        app: ai-agent
        version: v1.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9091"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: ai-agent
        image: myregistry/ai-agent-service:latest
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 9091
          name: metrics
          protocol: TCP
        env:
        - name: HELIOS_DB_PATH
          value: "/data/agent_cache.db"
        - name: HELIOS_CONFIG
          value: "/config/helios_agent_cache.toml"
        - name: AGENT_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: cache-data
          mountPath: /data
        - name: config
          mountPath: /config
          readOnly: true
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2
      volumes:
      - name: cache-data
        persistentVolumeClaim:
          claimName: agent-cache-pvc
      - name: config
        configMap:
          name: helios-agent-config
---
apiVersion: v1
kind: Service
metadata:
  name: ai-agent-service
  namespace: ai-platform
  labels:
    app: ai-agent
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  - port: 9091
    targetPort: 9091
    protocol: TCP
    name: metrics
  selector:
    app: ai-agent
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-cache-pvc
  namespace: ai-platform
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: helios-agent-config
  namespace: ai-platform
data:
  helios_agent_cache.toml: |
    [database]
    type = "embedded"
    path = "/data/agent_cache.db"
    mode = "readwrite-create"
    page_size = 4096
    cache_size_mb = 1024
    wal_mode = true

    [helios_proxy]
    enabled = true
    semantic_matching = true
    similarity_threshold = 0.92

    [cache_policies]
    default_ttl_seconds = 3600

    [cache_policies.tools]
    "weather.current" = 300
    "stock.price" = 60
    "company.info" = 86400

Example 4: Microservices Integration (Go/Rust)

Architecture Diagram:

                    ┌──────────────────────┐
                    │   API Gateway        │
                    │   (Kong/Nginx)       │
                    └──────────┬───────────┘
                               │
                ┌──────────────┴────────────────┐
                │                               │
        ┌───────▼────────┐            ┌────────▼────────┐
        │  AI Agent API  │            │  AI Agent API   │
        │   Instance 1   │            │   Instance 2    │
        │  ┌──────────┐  │            │  ┌──────────┐   │
        │  │ HeliosDB │  │            │  │ HeliosDB │   │
        │  │   Lite   │  │            │  │   Lite   │   │
        │  │ (Embedded)│  │            │  │ (Embedded)│  │
        │  └──────────┘  │            │  └──────────┘   │
        └────────────────┘            └─────────────────┘
              │                                 │
              └──────────────┬──────────────────┘
                             │
                    ┌────────▼────────┐
                    │   External APIs │
                    │ - Weather       │
                    │ - Stock Market  │
                    │ - Company DB    │
                    └─────────────────┘

Example 5: Edge Computing & IoT Deployment

Edge TOML Configuration (helios_edge_agent.toml):

[database]
type = "embedded"
path = "/data/edge_agent_cache.db"
mode = "readwrite-create"
page_size = 4096
cache_size_mb = 128  # Limited for edge device
wal_mode = true
sync_mode = "normal"  # Balance safety vs. performance

[helios_proxy]
enabled = true
semantic_matching = false  # Disabled for lower CPU usage
similarity_threshold = 0.95

[cache_policies]
default_ttl_seconds = 7200  # Longer TTL for edge (limited connectivity)
max_cache_entries = 10000
eviction_policy = "lru"

[cache_policies.tools]
"sensor.temperature" = 60      # 1 minute
"sensor.humidity" = 60          # 1 minute
"weather.forecast" = 1800       # 30 minutes
"device.status" = 300           # 5 minutes
"location.geocode" = 86400      # 24 hours (rarely changes)

[edge_sync]
enabled = true
sync_interval_seconds = 300
central_endpoint = "https://central.example.com/api/sync"
sync_on_connectivity_restore = true
batch_size = 100

[performance]
max_concurrent_queries = 10  # Limited for edge device
query_timeout_ms = 5000
enable_query_plan_cache = true

[storage]
max_db_size_mb = 500
auto_vacuum = true
vacuum_interval_hours = 24

Architecture Diagram:

┌─────────────────────────────────────────────────────────────┐
│                     Edge Device (Raspberry Pi)               │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Edge Agent Process                                    │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌────────────┐  │  │
│  │  │ Sensor I/O   │  │  AI Inference│  │  Control   │  │  │
│  │  │ - GPIO Read  │  │  - Local LLM │  │  Logic     │  │  │
│  │  │ - I2C/SPI    │  │  - TinyML    │  │            │  │  │
│  │  └──────┬───────┘  └──────┬───────┘  └─────┬──────┘  │  │
│  │         │                 │                 │         │  │
│  │         └─────────────────┼─────────────────┘         │  │
│  │                           ▼                           │  │
│  │  ┌────────────────────────────────────────────────┐   │  │
│  │  │        HeliosDB-Lite Embedded Cache            │   │  │
│  │  │  - Sensor readings cache                       │   │  │
│  │  │  - Weather forecast cache                      │   │  │
│  │  │  - Tool call results cache                     │   │  │
│  │  │  - Sync queue (offline resilience)             │   │  │
│  │  └────────────────────────────────────────────────┘   │  │
│  └───────────────────────────────────────────────────────┘  │
│                           │                                 │
│                           │ Periodic sync (when online)     │
│                           ▼                                 │
└───────────────────────────┼─────────────────────────────────┘
                            │
                            │ HTTPS
                            ▼
                ┌───────────────────────┐
                │   Central Server      │
                │   - Data aggregation  │
                │   - Analytics         │
                │   - Dashboard         │
                └───────────────────────┘

Adoption Strategy

Phase 1: Proof of Concept (Weeks 1-2)

1. Deploy HeliosDB Lite: Add caching layer with default TTL policies

Phase 2: Pilot Deployment (Weeks 3-6)

1. Expand to All Tools: Add caching for all external APIs (10-20 tools)
2. Tune TTL Policies: Optimize per-tool TTLs based on freshness requirements
3. Enable Semantic Matching: Deploy embedding model for fuzzy cache hits
4. Monitor & Alert: Set up Prometheus metrics + Grafana dashboards

Phase 3: Production Rollout (Weeks 7-12)

1. Multi-tenant Isolation: Deploy per-agent database files for isolation
2. Edge Deployment: Roll out to edge devices with offline sync
3. Policy Automation: Implement automatic TTL adjustment based on observed patterns

Conclusion

Traditional caching solutions—Redis clusters, cloud services, in-memory dictionaries—fail to address the unique requirements of AI workloads: semantic similarity matching, context-aware TTL policies, offline operation, and per-agent isolation.

The embedded architecture uniquely enables offline-first edge deployments, where agents can operate indefinitely without connectivity—critical for IoT, robotics, and field applications. As AI agents become the primary interface for enterprise applications, the ability to cache tool results efficiently and intelligently will be the difference between economically viable deployments and those that burn budget on redundant API calls.