HeliosDB Phase 1 Staging Deployment Guide

HeliosDB Phase 1 Staging Deployment Guide

Version: 7.0.0 Environment: Staging Features: Conversational BI, Auto-Compliance, Embedded+Cloud Unified Last Updated: 2025-11-17

---

Table of Contents

1. Overview
2. Prerequisites
3. Pre-Deployment Checklist
4. Deployment Methods
5. Configuration
6. Deployment Steps
7. Post-Deployment Validation
8. Troubleshooting

---

Overview

This guide provides step-by-step instructions for deploying HeliosDB Phase 1 production-ready features to the staging environment. The deployment includes:

• Conversational BI Service - Natural language to SQL with LLM integration
• Auto-Compliance Service - SOC2, GDPR, HIPAA, PCI-DSS, ISO27001 compliance automation
• Embedded+Cloud Sync Service - Unified embedded and cloud database with offline support

Architecture

┌─────────────────────────────────────────────────────────┐
│                Load Balancer (Nginx)                     │
│              HTTPS / Rate Limiting / Routing             │
└────────────────────┬────────────────────────────────────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
        ▼            ▼            ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│Conversational│ │Auto-Compliance│ │Embedded+Cloud│
│BI Service    │ │Service        │ │Sync Service  │
│:8081         │ │:8082 :8090    │ │:8083 :8084   │
└──────┬───────┘ └──────┬────────┘ └──────┬───────┘
       │                │                 │
       ▼                ▼                 ▼
┌──────────────────────────────────────────────┐
│         Shared Infrastructure                 │
│  PostgreSQL  │  Redis  │  Prometheus │Grafana │
└──────────────────────────────────────────────┘

---

Prerequisites

System Requirements

Minimum (Development/Testing):

• CPU: 8 cores
• RAM: 16 GB
• Disk: 100 GB SSD
• Network: 1 Gbps

Recommended (Staging):

• CPU: 16 cores
• RAM: 32 GB
• Disk: 500 GB SSD (NVMe preferred)
• Network: 10 Gbps

Software Dependencies

• Docker: >= 24.0.0
• Docker Compose: >= 2.20.0
• Kubernetes (optional): >= 1.28
• Git: >= 2.40.0
• curl: Latest
• OpenSSL: >= 3.0

External Services

Required:

1. LLM API Access (at least one):

   • OpenAI API key (GPT-4 recommended)
   • Anthropic API key (Claude 3.5 Sonnet)
   • Cohere API key

2. Cloud Storage (for Embedded+Cloud):

   • AWS S3 bucket + credentials, OR
   • S3-compatible storage (MinIO, DigitalOcean Spaces, etc.)

Optional:

• Email SMTP (for Compliance alerts)
• Monitoring Integration (PagerDuty, Slack, etc.)

---

Pre-Deployment Checklist

1. Infrastructure Preparation

• Provision compute resources (VMs, cloud instances, or Kubernetes cluster)
• Configure network security groups/firewall rules
• Set up DNS records (if using domain names)
• Provision persistent storage volumes
• Configure backup infrastructure

2. Credentials and Secrets

• Obtain LLM API keys (OpenAI/Anthropic/Cohere)
• Generate strong passwords for PostgreSQL and Redis
• Set up AWS/S3 credentials for cloud storage
• Configure SMTP credentials (if using email alerts)
• Create Grafana admin password

3. Code and Builds

• Clone HeliosDB repository
• Checkout v7.0.0 tag
• Verify all source code is present
• Review and customize configuration files

4. Environment Configuration

• Copy .env.example to .env
• Fill in all required environment variables
• Validate configuration file syntax
• Review resource limits and quotas

---

Deployment Methods

HeliosDB Phase 1 supports two primary deployment methods:

Method 1: Docker Compose (Recommended for Single-Node Staging)

Pros:

• Simple, single-command deployment
• Easier to debug and troubleshoot
• Lower resource overhead
• Fast iteration during testing

Cons:

• No high availability
• Limited scalability
• Manual load balancing

Use When:

• Deploying to a single server/VM
• Rapid testing and validation
• Development or staging environments

Method 2: Kubernetes (Recommended for Production-Like Staging)

Pros:

• High availability and self-healing
• Horizontal scalability
• Built-in service discovery and load balancing
• Production parity

Cons:

• Higher complexity
• More resource overhead
• Steeper learning curve

Use When:

• Multi-node deployment
• Production-like staging environment
• Need for high availability
• Load testing at scale

---

Configuration

Environment Variables

Copy the template and customize:

cd /home/claude/HeliosDB/deployment/staging
cp .env.example .env

Edit .env with your values:

# Database
POSTGRES_PASSWORD=<strong-random-password>
REDIS_PASSWORD=<strong-random-password>

# LLM APIs (at least one required)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
COHERE_API_KEY=...

# Cloud Storage (for Embedded+Cloud)
S3_BUCKET=heliosdb-sync-staging
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=AKIA...
S3_SECRET_ACCESS_KEY=...

# Compliance (optional email alerts)
ALERT_EMAIL_ENABLED=true
ALERT_EMAIL_SMTP_HOST=smtp.gmail.com
ALERT_EMAIL_SMTP_PORT=587
ALERT_EMAIL_FROM=compliance@yourdomain.com
ALERT_EMAIL_TO=admin@yourdomain.com

# Monitoring
GRAFANA_ADMIN_PASSWORD=<strong-random-password>

Validation

Validate your .env file:

# Check for missing required variables
grep -E "^(POSTGRES_PASSWORD|REDIS_PASSWORD|OPENAI_API_KEY|S3_BUCKET)=" .env || echo "Missing required variables!"

# Check for placeholder values
grep "changeme\|your-.*-here" .env && echo "WARNING: Found placeholder values!"

---

Deployment Steps

Method 1: Docker Compose Deployment

Step 1: Build Docker Images

cd /home/claude/HeliosDB

# Build all service images
docker compose -f deployment/staging/docker-compose.yml build

# Verify images were created
docker images | grep heliosdb

Expected output:

heliosdb-conversational-bi    latest    ...    ...    ...
heliosdb-compliance           latest    ...    ...    ...
heliosdb-embedded-cloud       latest    ...    ...    ...

Step 2: Start Infrastructure Services

# Start PostgreSQL and Redis first
docker compose -f deployment/staging/docker-compose.yml up -d postgres redis

# Wait for services to be healthy
docker compose -f deployment/staging/docker-compose.yml ps

# Check logs
docker compose -f deployment/staging/docker-compose.yml logs postgres redis

Step 3: Initialize Database

# Run database migrations (if applicable)
# docker compose -f deployment/staging/docker-compose.yml run --rm conversational-bi migrate

# Verify database is accessible
docker compose -f deployment/staging/docker-compose.yml exec postgres \
  psql -U heliosdb_admin -d heliosdb -c "SELECT version();"

Step 4: Start Feature Services

# Start all feature services
docker compose -f deployment/staging/docker-compose.yml up -d \
  conversational-bi \
  compliance \
  embedded-cloud-sync

# Wait for services to be healthy (this may take 1-2 minutes)
watch -n 2 'docker compose -f deployment/staging/docker-compose.yml ps'

Step 5: Start Monitoring Stack

# Start Prometheus and Grafana
docker compose -f deployment/staging/docker-compose.yml up -d prometheus grafana

# Verify Prometheus is scraping metrics
curl http://localhost:9090/api/v1/targets

Step 6: Start Load Balancer

# Start Nginx
docker compose -f deployment/staging/docker-compose.yml up -d nginx

# Verify Nginx is routing correctly
curl http://localhost/health

Step 7: Verify All Services

# Check all containers are running
docker compose -f deployment/staging/docker-compose.yml ps

# Expected: All services in "Up" state and healthy

Method 2: Kubernetes Deployment

Step 1: Prepare Kubernetes Cluster

# Verify kubectl is configured
kubectl cluster-info

# Create namespace
kubectl apply -f /home/claude/HeliosDB/deployment/staging/kubernetes/00-namespace.yaml

# Verify namespace
kubectl get namespace heliosdb-staging

Step 2: Create Secrets

# Create secrets from .env file
kubectl create secret generic heliosdb-secrets \
  --from-env-file=/home/claude/HeliosDB/deployment/staging/.env \
  --namespace=heliosdb-staging

# Verify secret
kubectl get secrets -n heliosdb-staging

Step 3: Deploy Infrastructure

# Deploy PostgreSQL
kubectl apply -f deployment/staging/kubernetes/postgres-statefulset.yaml

# Deploy Redis
kubectl apply -f deployment/staging/kubernetes/redis-deployment.yaml

# Wait for ready
kubectl wait --for=condition=ready pod -l app=postgres -n heliosdb-staging --timeout=300s
kubectl wait --for=condition=ready pod -l app=redis -n heliosdb-staging --timeout=300s

Step 4: Deploy Feature Services

# Deploy Conversational BI
kubectl apply -f deployment/staging/kubernetes/conversational-bi-deployment.yaml

# Deploy Compliance
kubectl apply -f deployment/staging/kubernetes/compliance-deployment.yaml

# Deploy Embedded+Cloud
kubectl apply -f deployment/staging/kubernetes/embedded-cloud-deployment.yaml

# Wait for ready
kubectl wait --for=condition=available deployment --all -n heliosdb-staging --timeout=600s

Step 5: Deploy Monitoring

# Deploy Prometheus
kubectl apply -f deployment/staging/kubernetes/prometheus-deployment.yaml

# Deploy Grafana
kubectl apply -f deployment/staging/kubernetes/grafana-deployment.yaml

# Wait for ready
kubectl wait --for=condition=ready pod -l app=prometheus -n heliosdb-staging --timeout=300s
kubectl wait --for=condition=ready pod -l app=grafana -n heliosdb-staging --timeout=300s

Step 6: Deploy Ingress

# Deploy Ingress controller (if not already installed)
kubectl apply -f deployment/staging/kubernetes/ingress.yaml

# Get Ingress IP
kubectl get ingress -n heliosdb-staging

---

Post-Deployment Validation

1. Service Health Checks

Docker Compose:

# Check service health
curl http://localhost:8081/health  # Conversational BI
curl http://localhost:8082/health  # Compliance
curl http://localhost:8083/health  # Embedded+Cloud

# All should return: {"status": "healthy"}

Kubernetes:

# Check pod status
kubectl get pods -n heliosdb-staging

# Check service endpoints
kubectl get endpoints -n heliosdb-staging

# Test health checks
kubectl exec -it -n heliosdb-staging deployment/conversational-bi -- curl http://localhost:8081/health

2. Smoke Tests

Conversational BI:

# Test NL2SQL endpoint
curl -X POST http://localhost:8081/api/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "Show me all users created in the last 7 days",
    "database": "heliosdb"
  }'

# Expected: Valid SQL query response

Compliance:

# Test compliance check endpoint
curl http://localhost:8082/api/v1/compliance/status

# Expected: Compliance framework status

Embedded+Cloud:

# Test sync endpoint
curl http://localhost:8083/api/v1/sync/status

# Expected: Sync service status

3. Monitoring Validation

Prometheus:

# Access Prometheus UI
open http://localhost:9090

# Verify targets are up
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# Expected: All targets showing "up"

Grafana:

# Access Grafana UI
open http://localhost:3000

# Login with credentials from .env
# Username: admin
# Password: <GRAFANA_ADMIN_PASSWORD>

# Verify dashboards:
# - HeliosDB Conversational BI - Production Metrics
# - HeliosDB Auto-Compliance - Production Metrics
# - HeliosDB Embedded+Cloud Unified - Production Metrics

4. Metrics Validation

# Check Conversational BI metrics
curl http://localhost:9091/metrics | grep heliosdb_conversational_bi

# Check Compliance metrics
curl http://localhost:9092/metrics | grep heliosdb_compliance

# Check Embedded+Cloud metrics
curl http://localhost:9093/metrics | grep heliosdb_embedded_cloud

5. Log Validation

Docker Compose:

# View service logs
docker compose -f deployment/staging/docker-compose.yml logs -f conversational-bi
docker compose -f deployment/staging/docker-compose.yml logs -f compliance
docker compose -f deployment/staging/docker-compose.yml logs -f embedded-cloud-sync

# Check for errors
docker compose -f deployment/staging/docker-compose.yml logs | grep -i error

Kubernetes:

# View service logs
kubectl logs -f -n heliosdb-staging deployment/conversational-bi
kubectl logs -f -n heliosdb-staging deployment/compliance
kubectl logs -f -n heliosdb-staging deployment/embedded-cloud-sync

# Check for errors
kubectl logs -n heliosdb-staging --all-containers=true | grep -i error

---

Troubleshooting

Common Issues

Issue 1: Service Won’t Start

Symptoms:

• Container exits immediately
• Health check fails
• “Unhealthy” status

Diagnosis:

# Check logs
docker compose -f deployment/staging/docker-compose.yml logs <service-name>

# Check container status
docker compose -f deployment/staging/docker-compose.yml ps

Common Causes:

1. Missing environment variables
2. Database connection failure
3. Port conflict
4. Insufficient resources

Solutions:

# 1. Validate environment variables
docker compose -f deployment/staging/docker-compose.yml config

# 2. Test database connectivity
docker compose -f deployment/staging/docker-compose.yml exec <service> \
  nc -zv postgres 5432

# 3. Check port availability
ss -tulpn | grep <port>

# 4. Check resource usage
docker stats

Issue 2: High Memory Usage

Symptoms:

• Container OOM kills
• Slow performance
• Service restarts

Diagnosis:

# Check memory usage
docker stats --no-stream

# Check container limits
docker inspect <container-id> | jq '.[].HostConfig.Memory'

Solutions:

# Increase memory limits in docker-compose.yml
# Under deploy.resources.limits.memory

# Restart service with new limits
docker compose -f deployment/staging/docker-compose.yml up -d --force-recreate <service>

Issue 3: LLM API Errors

Symptoms:

• “API key invalid” errors
• High latency
• Rate limit errors

Diagnosis:

# Check Conversational BI logs
docker compose -f deployment/staging/docker-compose.yml logs conversational-bi | grep -i "llm\|api"

# Check metrics
curl http://localhost:9091/metrics | grep llm_errors

Solutions:

1. Verify API keys are correct
2. Check API rate limits
3. Switch to alternate LLM provider
4. Increase rate limit settings

Issue 4: Database Connection Issues

Symptoms:

• “Connection refused”
• “Too many connections”
• Timeout errors

Diagnosis:

# Check PostgreSQL is running
docker compose -f deployment/staging/docker-compose.yml ps postgres

# Check PostgreSQL logs
docker compose -f deployment/staging/docker-compose.yml logs postgres

# Check connection count
docker compose -f deployment/staging/docker-compose.yml exec postgres \
  psql -U heliosdb_admin -d heliosdb -c "SELECT count(*) FROM pg_stat_activity;"

Solutions:

# 1. Restart PostgreSQL
docker compose -f deployment/staging/docker-compose.yml restart postgres

# 2. Increase max connections (in postgres config)
# Edit deployment/staging/configs/postgres/postgresql.conf
# max_connections = 200

# 3. Restart all services
docker compose -f deployment/staging/docker-compose.yml restart

Issue 5: Monitoring Not Working

Symptoms:

• Grafana shows “No data”
• Prometheus can’t reach targets
• Metrics not updating

Diagnosis:

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'

# Check Prometheus logs
docker compose -f deployment/staging/docker-compose.yml logs prometheus

# Check service metrics endpoints
curl http://localhost:9091/metrics  # Conversational BI
curl http://localhost:9092/metrics  # Compliance
curl http://localhost:9093/metrics  # Embedded+Cloud

Solutions:

1. Verify Prometheus configuration
2. Check network connectivity between services
3. Restart monitoring stack
4. Verify firewall rules

---

Next Steps

After successful deployment:

1. Review Monitoring: Check Grafana dashboards for baseline metrics
2. Configure Alerts: Set up alert notifications (see MONITORING_GUIDE.md)
3. Load Testing: Run validation tests (see VALIDATION_TEST_SUITE.md)
4. Documentation: Update deployment notes with any customizations
5. Backups: Configure automated backups (see BACKUP_PROCEDURES.md)

---

Support and Resources

• Deployment Issues: See TROUBLESHOOTING.md
• Rollback Procedures: See ROLLBACK_PROCEDURES.md
• Incident Response: See INCIDENT_RESPONSE.md
• Performance Tuning: See PERFORMANCE_TUNING.md
• Monitoring Guide: See MONITORING_GUIDE.md

---

Deployment Complete! Your HeliosDB Phase 1 staging environment is now operational.