---
title: "GPU Operations Runbook"
---

# HeliosDB GPU Operations Runbook

:::caution[GPU acceleration: builds on request]
GPU acceleration (NVIDIA CUDA) is available in HeliosDB Full builds on request and
is not enabled in the public release build; AMD ROCm support is planned. In
the public release build, queries run on the CPU and the GPU settings and commands on
this page have no effect. Contact [sales@heliosdb.com](mailto:sales@heliosdb.com)
about a CUDA-enabled build.
:::

> **Container images:** HeliosDB Full images are not published on Docker Hub or any other public registry. Images and Helm charts are provided through private registry access during onboarding: contact [sales@heliosdb.com](mailto:sales@heliosdb.com). The examples on this page use `heliosdb-full:<tag>`, an image built locally from the licensed source distribution (`docker build -t heliosdb-full:<tag> .`); replace it with the image reference you receive if you use the private registry.

Use this runbook for GPU acceleration enablement, health checks, fallback, and troubleshooting.

## Enable/Disable GPU Acceleration

Before enabling GPU acceleration:

- Confirm supported GPU hardware and drivers.
- Confirm CUDA or ROCm libraries are installed.
- Validate HeliosDB configuration.
- Run a representative query benchmark in staging.

Configuration checks:

```bash
nvidia-smi
rocm-smi
heliosdb --gpu-check
heliosdb --config /etc/heliosdb/heliosdb.toml --validate-config
```

Disable GPU acceleration when hardware health is degraded, GPU memory is exhausted, or CPU fallback gives more stable latency.

## GPU Health Monitoring

Monitor:

- Device availability.
- GPU temperature.
- Memory usage.
- Error counters.
- Kernel execution failures.
- Query fallback rate.

Example:

```bash
nvidia-smi --query-gpu=name,temperature.gpu,memory.used,memory.total,utilization.gpu --format=csv
curl -s http://heliosdb-node:7000/metrics | grep gpu
```

## GPU Memory Management

If GPU memory pressure is high:

1. Identify GPU-backed query classes.
2. Lower batch size or concurrency.
3. Raise CPU fallback threshold for small queries.
4. Clear abandoned sessions.
5. Restart only if memory is leaked and fallback is not sufficient.

## Fallback to CPU Procedure

1. Disable GPU scheduling for new queries.
2. Let in-flight GPU queries finish or cancel long-running work.
3. Confirm CPU execution is active.
4. Monitor latency and CPU pressure.
5. Record the reason for fallback.

## GPU Troubleshooting

| Symptom | Check | Action |
|---|---|---|
| GPU not detected | Driver and container device access | Restart driver path or redeploy node |
| GPU out of memory | Query concurrency and batch size | Reduce concurrency or fallback |
| GPU errors | Device logs and ECC counters | Remove device from serving pool |
| Slow GPU query | Dataset size and transfer overhead | Use CPU for small workloads |

## CUDA/ROCm Diagnostics

Collect diagnostics before replacing hardware:

```bash
nvidia-smi -q
nvcc --version
rocm-smi --showallinfo
/opt/rocm/bin/rocminfo
```