Circuit Breaker for Automatic Failover Protection

Problem Being Solved

Core Problem Statement

Traditional database connection pools and load balancers lack intelligent failure detection and isolation mechanisms, treating all connection errors identically and employing naive retry logic that amplifies problems. When a backend degrades (slow queries, high CPU, storage saturation), connection pools continue sending traffic, timeouts accumulate, retry storms begin, and the load redistributes to healthy backends in an uncontrolled manner—often overwhelming them and creating a cascading failure. By the time operations teams detect and respond, multiple systems are impacted and recovery requires manual intervention.

Concrete Examples with Code, Config & Architecture

Example 1: Embedded Configuration for Circuit Breaker

Configuration: helios_circuit_breaker.toml

[proxy]
listen_address = "0.0.0.0:5432"
protocol = "postgresql"
circuit_breaker_enabled = true

[circuit_breaker]
# Core circuit breaker behavior
enabled = true
mode = "adaptive"  # "adaptive" | "static" | "predictive"

# Failure detection thresholds
failure_threshold_percentage = 5.0     # 5% error rate triggers circuit open
latency_threshold_p95_ms = 500
latency_threshold_p99_ms = 2000
connection_saturation_threshold = 0.95 # 95% pool utilization triggers circuit open

# Sliding window for metrics aggregation
metrics_window_size = "10s"            # Evaluate metrics over 10-second window
metrics_bucket_size = "1s"             # 1-second granularity for aggregation
minimum_requests = 20                  # Minimum requests before circuit can trip

# Circuit state transition timing
open_duration = "30s"                  # How long to wait before testing recovery
half_open_max_concurrent = 10          # Max concurrent probe requests in HALF_OPEN
half_open_success_threshold = 80       # 80% success rate to close circuit
half_open_probe_interval = "5s"        # Interval between probe attempts

# Recovery behavior
gradual_recovery_enabled = true
recovery_ramp_up_duration = "120s"     # 2 minutes to fully restore traffic
recovery_step_percentage = 10          # Increase traffic by 10% per step

# Advanced: Query-level circuit breakers
query_level_breakers_enabled = true
slow_query_threshold_ms = 5000
query_pattern_detection = true         # Automatically detect problematic query patterns

# Advanced: User-level circuit breakers (prevent noisy neighbor)
user_level_breakers_enabled = true
per_user_rate_limit = 1000             # Max 1000 queries/sec per user
user_isolation_on_abuse = true         # Auto-isolate abusive users

[backends]
# Backend 1: Primary
[[backends.instances]]
name = "primary-1"
host = "db-primary-1.internal"
port = 5432
priority = 100
weight = 1.0

# Circuit breaker overrides for this backend
circuit_breaker_latency_threshold_p95_ms = 300  # Stricter for primary
circuit_breaker_failure_threshold = 3.0         # Lower tolerance

# Backend 2: Primary
[[backends.instances]]
name = "primary-2"
host = "db-primary-2.internal"
port = 5432
priority = 100
weight = 1.0

# Backend 3: Analytics replica (more tolerant)
[[backends.instances]]
name = "analytics-1"
host = "db-analytics-1.internal"
port = 5432
priority = 50
weight = 0.5

circuit_breaker_latency_threshold_p95_ms = 2000  # Analytics can be slower
circuit_breaker_failure_threshold = 10.0         # Higher tolerance
allowed_query_types = ["SELECT"]                 # Read-only

[health_checks]
# Active health check configuration
enabled = true
interval = "1s"                        # Check every second
timeout = "500ms"
check_query = "SELECT 1"               # Simple liveness check

# Comprehensive health metrics
collect_latency_percentiles = true
collect_connection_metrics = true
collect_replication_lag = true
collect_query_queue_depth = true

[observability]
# Monitoring and alerting
metrics_enabled = true
metrics_port = 9090
log_circuit_state_changes = true
log_level = "info"

# Prometheus metrics exposed:
# - helios_circuit_breaker_state{backend} (0=closed, 1=open, 2=half_open)
# - helios_circuit_breaker_failures_total{backend}
# - helios_circuit_breaker_trip_duration_seconds{backend}
# - helios_backend_latency_p95_milliseconds{backend}
# - helios_backend_error_rate{backend}
# - helios_backend_connection_saturation{backend}

[alerts]
# Automatic alerting (optional integration)
slack_webhook_url = "${SLACK_WEBHOOK_URL}"
pagerduty_integration_key = "${PAGERDUTY_KEY}"

alert_on_circuit_open = true
alert_on_multiple_circuits_open = true
alert_on_recovery_failure = true

Example 3: Infrastructure & Container Deployment

Docker Compose with Circuit Breaker Configuration:

version: '3.9'

services:
  # HeliosProxy with circuit breaker (front-end)
  heliosproxy:
    image: ghcr.io/heliosdatabase/hdb-heliosdb-proxy:<version>
    container_name: heliosproxy
    hostname: proxy.internal
    environment:
      HELIOS_CONFIG: /etc/helios/helios_circuit_breaker.toml
      HELIOS_LOG_LEVEL: info
      RUST_BACKTRACE: 1
    volumes:
      - ./helios_circuit_breaker.toml:/etc/helios/helios_circuit_breaker.toml:ro
      - helios-proxy-logs:/var/log/helios
    networks:
      - helios-network
    ports:
      - "5432:5432"  # PostgreSQL protocol
      - "9090:9090"  # Prometheus metrics
      - "8080:8080"  # Circuit breaker status API
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 5s
      timeout: 3s
      retries: 3
    deploy:
      resources:
        limits:
          memory: 4G
          cpus: '2.0'
        reservations:
          memory: 2G
          cpus: '1.0'

  # Backend 1: Primary database
  heliosdb-backend-1:
    image: registry.example.com/heliosdb-lite:<version>  # licensed image from HeliosDB sales, or built from the Lite Dockerfile
    container_name: heliosdb-backend-1
    hostname: db-backend-1.internal
    environment:
      HELIOS_MODE: primary
      POSTGRES_USER: helios
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ecommerce
    volumes:
      - helios-backend-1-data:/var/lib/postgresql/data
    networks:
      - helios-network
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "helios"]
      interval: 1s
      timeout: 500ms
      retries: 3

  # Backend 2: Primary database
  heliosdb-backend-2:
    image: registry.example.com/heliosdb-lite:<version>
    container_name: heliosdb-backend-2
    hostname: db-backend-2.internal
    environment:
      HELIOS_MODE: primary
      POSTGRES_USER: helios
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ecommerce
    volumes:
      - helios-backend-2-data:/var/lib/postgresql/data
    networks:
      - helios-network
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "helios"]
      interval: 1s
      timeout: 500ms
      retries: 3

  # Backend 3: Primary database
  heliosdb-backend-3:
    image: registry.example.com/heliosdb-lite:<version>
    container_name: heliosdb-backend-3
    hostname: db-backend-3.internal
    environment:
      HELIOS_MODE: primary
      POSTGRES_USER: helios
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ecommerce
    volumes:
      - helios-backend-3-data:/var/lib/postgresql/data
    networks:
      - helios-network
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "helios"]
      interval: 1s
      timeout: 500ms
      retries: 3

  # Application: E-commerce checkout service
  checkout-service:
    build:
      context: ./checkout-service
      dockerfile: Dockerfile
    container_name: checkout-service
    environment:
      DATABASE_URL: postgresql://helios:${DB_PASSWORD}@proxy.internal:5432/ecommerce
      PORT: 3000
    networks:
      - helios-network
    ports:
      - "3000:3000"
    depends_on:
      heliosproxy:
        condition: service_healthy
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        delay: 10s

  # Chaos engineering: Simulate backend failures
  chaos-monkey:
    image: chaos-engineering/chaos-monkey:latest
    container_name: chaos-monkey
    environment:
      TARGETS: heliosdb-backend-1,heliosdb-backend-2,heliosdb-backend-3
      FAILURE_MODE: latency  # Inject latency spikes
      FAILURE_INTERVAL: 300s # Every 5 minutes
      FAILURE_DURATION: 120s # 2 minutes of degradation
    networks:
      - helios-network
    depends_on:
      - heliosdb-backend-1
      - heliosdb-backend-2
      - heliosdb-backend-3

  # Monitoring: Prometheus
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    networks:
      - helios-network
    ports:
      - "9091:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'

  # Monitoring: Grafana with circuit breaker dashboard
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
      GF_INSTALL_PLUGINS: grafana-piechart-panel,grafana-clock-panel
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-dashboards/circuit-breaker.json:/etc/grafana/provisioning/dashboards/circuit-breaker.json:ro
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro
    networks:
      - helios-network
    ports:
      - "3001:3000"
    depends_on:
      - prometheus

networks:
  helios-network:
    driver: bridge

volumes:
  helios-backend-1-data:
  helios-backend-2-data:
  helios-backend-3-data:
  helios-proxy-logs:
  prometheus-data:
  grafana-data:

Kubernetes Deployment with Circuit Breaker:

apiVersion: v1
kind: ConfigMap
metadata:
  name: heliosproxy-circuit-breaker-config
  namespace: production
data:
  helios_circuit_breaker.toml: |
    [proxy]
    listen_address = "0.0.0.0:5432"
    circuit_breaker_enabled = true

    [circuit_breaker]
    failure_threshold_percentage = 5.0
    latency_threshold_p95_ms = 500
    open_duration = "30s"
    gradual_recovery_enabled = true

    [backends]
    [[backends.instances]]
    name = "backend-1"
    host = "heliosdb-backend-1.production.svc.cluster.local"
    port = 5432
    priority = 100

    [[backends.instances]]
    name = "backend-2"
    host = "heliosdb-backend-2.production.svc.cluster.local"
    port = 5432
    priority = 100

    [[backends.instances]]
    name = "backend-3"
    host = "heliosdb-backend-3.production.svc.cluster.local"
    port = 5432
    priority = 100

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: heliosproxy
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: heliosproxy
  template:
    metadata:
      labels:
        app: heliosproxy
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: heliosproxy
        image: ghcr.io/heliosdatabase/hdb-heliosdb-proxy:<version>
        ports:
        - containerPort: 5432
          name: postgres
        - containerPort: 9090
          name: metrics
        - containerPort: 8080
          name: status-api
        volumeMounts:
        - name: config
          mountPath: /etc/helios/helios_circuit_breaker.toml
          subPath: helios_circuit_breaker.toml
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3
      volumes:
      - name: config
        configMap:
          name: heliosproxy-circuit-breaker-config

---
apiVersion: v1
kind: Service
metadata:
  name: heliosproxy
  namespace: production
spec:
  selector:
    app: heliosproxy
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432
    - name: metrics
      port: 9090
      targetPort: 9090
  type: ClusterIP

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: heliosproxy-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: heliosproxy

Example 4: Microservices Integration (Go/Rust)

Go Microservice with Circuit Breaker Awareness:

package main

import (
    "context"
    "database/sql"
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "time"

    "github.com/gin-gonic/gin"
    _ "github.com/lib/pq"
)

type OrderService struct {
    db *sql.DB
}

type CircuitBreakerStatus struct {
    Backend              string  `json:"backend"`
    CircuitState         string  `json:"circuit_state"`
    LatencyP95Ms         float64 `json:"latency_p95_ms"`
    ErrorRate            float64 `json:"error_rate"`
    ConnectionSaturation float64 `json:"connection_saturation"`
}

func main() {
    // Connect to HeliosDB-Lite via HeliosProxy
    // Circuit breaker operates transparently at proxy level
    dsn := "postgres://order_service:password@heliosproxy:5432/ecommerce?sslmode=disable"
    db, err := sql.Open("postgres", dsn)
    if err != nil {
        log.Fatalf("Failed to connect to database: %v", err)
    }
    defer db.Close()

    // Configure connection pool
    db.SetMaxOpenConns(50)
    db.SetMaxIdleConns(10)
    db.SetConnMaxLifetime(time.Hour)

    service := &OrderService{db: db}

    // Initialize Gin router
    router := gin.Default()

    // API endpoints
    router.POST("/api/v1/orders", service.CreateOrder)
    router.GET("/api/v1/orders/:id", service.GetOrder)
    router.GET("/api/v1/health", service.HealthCheck)
    router.GET("/api/v1/circuit-breaker/status", service.GetCircuitBreakerStatus)

    log.Println("Order Service starting on :8080")
    log.Println("Circuit breaker protection: active (via HeliosProxy)")
    router.Run(":8080")
}

func (s *OrderService) CreateOrder(c *gin.Context) {
    var request struct {
        UserID  string `json:"user_id"`
        Items   []struct {
            ProductID string  `json:"product_id"`
            Quantity  int     `json:"quantity"`
            Price     float64 `json:"price"`
        } `json:"items"`
    }

    if err := c.ShouldBindJSON(&request); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    startTime := time.Now()

    // Start transaction
    // If backend fails during this transaction, circuit breaker will:
    // 2. Open circuit for failed backend
    // 3. Route subsequent queries to healthy backend
    // 4. Return retryable error to client
    ctx := context.Background()
    tx, err := s.db.BeginTx(ctx, nil)
    if err != nil {
        log.Printf("Failed to begin transaction: %v", err)
        c.JSON(http.StatusServiceUnavailable, gin.H{
            "error":     "Database temporarily unavailable",
            "retryable": true, // Circuit breaker makes retries safe
        })
        return
    }
    defer tx.Rollback()

    // Calculate total
    var totalAmount float64
    for _, item := range request.Items {
        totalAmount += item.Price * float64(item.Quantity)
    }

    // Insert order
    var orderID string
    err = tx.QueryRowContext(ctx, `
        INSERT INTO orders (user_id, total_amount, status, created_at)
        VALUES ($1, $2, 'PENDING', NOW())
        RETURNING order_id
    `, request.UserID, totalAmount).Scan(&orderID)

    if err != nil {
        log.Printf("Failed to create order: %v", err)
        c.JSON(http.StatusInternalServerError, gin.H{
            "error":     "Failed to create order",
            "retryable": true,
        })
        return
    }

    // Insert order items
    for _, item := range request.Items {
        _, err = tx.ExecContext(ctx, `
            INSERT INTO order_items (order_id, product_id, quantity, price)
            VALUES ($1, $2, $3, $4)
        `, orderID, item.ProductID, item.Quantity, item.Price)

        if err != nil {
            log.Printf("Failed to insert order item: %v", err)
            c.JSON(http.StatusInternalServerError, gin.H{
                "error":     "Failed to create order items",
                "retryable": true,
            })
            return
        }
    }

    // Commit transaction
    if err = tx.Commit(); err != nil {
        log.Printf("Failed to commit transaction: %v", err)
        c.JSON(http.StatusInternalServerError, gin.H{
            "error":     "Failed to commit order",
            "retryable": true,
        })
        return
    }

    elapsed := time.Since(startTime)

    log.Printf("Order created: order_id=%s, amount=%.2f, items=%d, time=%dms",
        orderID, totalAmount, len(request.Items), elapsed.Milliseconds())

    c.JSON(http.StatusCreated, gin.H{
        "order_id":           orderID,
        "total_amount":       totalAmount,
        "status":             "PENDING",
        "processing_time_ms": elapsed.Milliseconds(),
    })
}

func (s *OrderService) GetOrder(c *gin.Context) {
    orderID := c.Param("id")

    var order struct {
        OrderID     string    `json:"order_id"`
        UserID      string    `json:"user_id"`
        TotalAmount float64   `json:"total_amount"`
        Status      string    `json:"status"`
        CreatedAt   time.Time `json:"created_at"`
    }

    err := s.db.QueryRow(`
        SELECT order_id, user_id, total_amount, status, created_at
        FROM orders
        WHERE order_id = $1
    `, orderID).Scan(
        &order.OrderID,
        &order.UserID,
        &order.TotalAmount,
        &order.Status,
        &order.CreatedAt,
    )

    if err == sql.ErrNoRows {
        c.JSON(http.StatusNotFound, gin.H{"error": "Order not found"})
        return
    }

    if err != nil {
        log.Printf("Failed to fetch order: %v", err)
        c.JSON(http.StatusInternalServerError, gin.H{
            "error":     "Failed to fetch order",
            "retryable": true,
        })
        return
    }

    c.JSON(http.StatusOK, order)
}

func (s *OrderService) HealthCheck(c *gin.Context) {
    // Check database connectivity
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()

    err := s.db.PingContext(ctx)
    if err != nil {
        c.JSON(http.StatusServiceUnavailable, gin.H{
            "status":  "unhealthy",
            "message": "Database unavailable",
        })
        return
    }

    c.JSON(http.StatusOK, gin.H{
        "status":  "healthy",
        "message": "Service operational",
    })
}

func (s *OrderService) GetCircuitBreakerStatus(c *gin.Context) {
    // Query circuit breaker status from HeliosProxy
    rows, err := s.db.Query(`
        SELECT
            backend_name,
            circuit_state,
            latency_p95_ms,
            error_rate,
            connection_saturation
        FROM helios_proxy.circuit_breaker_status
        ORDER BY backend_name
    `)

    if err != nil {
        log.Printf("Failed to fetch circuit breaker status: %v", err)
        c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to fetch status"})
        return
    }
    defer rows.Close()

    var statuses []CircuitBreakerStatus
    for rows.Next() {
        var status CircuitBreakerStatus
        err := rows.Scan(
            &status.Backend,
            &status.CircuitState,
            &status.LatencyP95Ms,
            &status.ErrorRate,
            &status.ConnectionSaturation,
        )
        if err != nil {
            log.Printf("Failed to scan row: %v", err)
            continue
        }
        statuses = append(statuses, status)
    }

    c.JSON(http.StatusOK, gin.H{
        "backends": statuses,
        "timestamp": time.Now(),
    })
}

Example 5: Edge Computing & IoT Deployment

Edge Device with Circuit Breaker (Manufacturing Sensor Network):

# Circuit breaker protects against local database failures

[helios]
mode = "edge"
data_dir = "/opt/helios/data"

[proxy]
listen_address = "127.0.0.1:5432"
circuit_breaker_enabled = true

[circuit_breaker]
# Edge-optimized thresholds
failure_threshold_percentage = 10.0    # More tolerant for edge
latency_threshold_p95_ms = 1000        # Edge can be slower
open_duration = "60s"                  # Longer recovery window

# Edge-specific: Handle intermittent connectivity
connection_failure_threshold = 3       # Requires 3 consecutive failures
recovery_probe_timeout = "5s"          # More patient with edge networks

[edge]
local_processing = true
cloud_sync_enabled = true
offline_mode_enabled = true

# Circuit breaker for cloud connectivity
cloud_circuit_breaker_enabled = true
cloud_failure_threshold = 5
cloud_open_duration = "300s"           # 5 minutes before retry

[backends]
# Local embedded instance (primary)
[[backends.instances]]
name = "local-primary"
host = "localhost"
port = 5433
priority = 100

# Local standby instance
[[backends.instances]]
name = "local-standby"
host = "localhost"
port = 5434
priority = 50

# Cloud instance (optional, for analytics)
[[backends.instances]]
name = "cloud-analytics"
host = "cloud.manufacturing.example.com"
port = 5432
priority = 10
allowed_query_types = ["SELECT"]       # Read-only queries to cloud
circuit_breaker_latency_threshold_p95_ms = 5000  # Very tolerant

Adoption Strategy

Phase 1: Pilot with Non-Critical Workload (Weeks 1-2)

Objective: Validate circuit breaker behavior in production with low-risk application

Steps: 1. Deploy HeliosProxy with circuit breaker enabled for staging environment 2. Configure conservative thresholds (high tolerance for latency/errors) 3. Simulate backend failures using chaos engineering tools 4. Observe circuit breaker behavior: detection speed, isolation, recovery 5. Tune thresholds based on workload characteristics

Phase 2: Production Rollout for Critical Services (Weeks 3-6)

Objective: Deploy circuit breaker protection for customer-facing applications

Steps: Monitor circuit breaker metrics alongside existing observability tools Conduct controlled failure tests during low-traffic periods 5. Create runbooks and alerting for circuit breaker events

Phase 3: Advanced Features & Optimization

Objective: Maximize value from circuit breaker; enable predictive capabilities

Steps: 1. Enable query-level and user-level circuit breakers for surgical isolation 2. Implement predictive failure detection (machine learning models) 3. Integrate circuit breaker with incident management (PagerDuty, Slack) 4. Tune gradual recovery parameters for optimal performance 5. Document cost savings and operational improvements for business case

Conclusion

The circuit breaker pattern, when implemented correctly at the database proxy layer, represents a fundamental shift from reactive incident response to proactive failure prevention.

The business case is compelling and immediate. E-commerce platforms protect revenue during peak periods. SaaS platforms reduce customer impact and SLA credits. Financial services meet regulatory resilience requirements. The common thread is risk reduction: the elimination of manual intervention from the critical path of failure recovery, and the prevention of localized failures from becoming system-wide outages.

Deep PostgreSQL protocol integration enables query-level circuit breakers and session migration integration that external proxies cannot replicate. Multi-dimensional health scoring and adaptive load redistribution prevent both false positives (unnecessary failovers) and false negatives (undetected degradation).