GPU Operations Runbook

HeliosDB GPU Operations Runbook

Use this runbook for GPU acceleration enablement, health checks, fallback, and troubleshooting.

Enable/Disable GPU Acceleration

Before enabling GPU acceleration:

Confirm supported GPU hardware and drivers.
Confirm CUDA or ROCm libraries are installed.
Validate HeliosDB configuration.
Run a representative query benchmark in staging.

Configuration checks:

nvidia-smi
rocm-smi
heliosdb --gpu-check
heliosdb --config /etc/heliosdb/heliosdb.toml --validate-config

Disable GPU acceleration when hardware health is degraded, GPU memory is exhausted, or CPU fallback gives more stable latency.

GPU Health Monitoring

Monitor:

Device availability.
GPU temperature.
Memory usage.
Error counters.
Kernel execution failures.
Query fallback rate.

Example:

nvidia-smi --query-gpu=name,temperature.gpu,memory.used,memory.total,utilization.gpu --format=csv
curl -s http://heliosdb-node:7000/metrics | grep gpu

GPU Memory Management

If GPU memory pressure is high:

Identify GPU-backed query classes.
Lower batch size or concurrency.
Raise CPU fallback threshold for small queries.
Clear abandoned sessions.
Restart only if memory is leaked and fallback is not sufficient.

Fallback to CPU Procedure

Disable GPU scheduling for new queries.
Let in-flight GPU queries finish or cancel long-running work.
Confirm CPU execution is active.
Monitor latency and CPU pressure.
Record the reason for fallback.

GPU Troubleshooting

Symptom	Check	Action
GPU not detected	Driver and container device access	Restart driver path or redeploy node
GPU out of memory	Query concurrency and batch size	Reduce concurrency or fallback
GPU errors	Device logs and ECC counters	Remove device from serving pool
Slow GPU query	Dataset size and transfer overhead	Use CPU for small workloads

CUDA/ROCm Diagnostics

Collect diagnostics before replacing hardware:

nvidia-smi -q
nvcc --version
rocm-smi --showallinfo
/opt/rocm/bin/rocminfo