Skip to content
HeliosDB HeliosDB Full

Scaling Operations Runbook

Use this runbook when capacity pressure, growth planning, or cost optimization requires changing cluster resources.

Manual Scaling Procedures

Vertical Scaling

Use vertical scaling when a node is CPU, memory, or I/O constrained and the workload cannot be redistributed quickly.

  1. Confirm the bottleneck with metrics.
  2. Drain one node from traffic.
  3. Stop the node cleanly.
  4. Resize the instance, container limits, or storage.
  5. Start the node and validate health.
  6. Return it to service and repeat gradually.

Horizontal Scaling

Use horizontal scaling when adding read capacity, shard capacity, or regional redundancy.

  1. Provision the new node with matching configuration.
  2. Join it to the cluster.
  3. Wait for bootstrap, sync, or shard assignment.
  4. Validate health and replication state.
  5. Add the node to serving traffic.

Auto-Scaling Configuration

Auto-scaling policies should use stable signals:

  • CPU utilization sustained above threshold.
  • Memory pressure sustained above threshold.
  • Queue depth or connection saturation.
  • Query latency above service objective.
  • Storage utilization and growth rate.

Avoid scaling on short spikes. Use cooldown periods and minimum healthy replica counts.

Resource Monitoring

Check current pressure:

Terminal window
curl -s http://heliosdb-node:7000/metrics | grep -E "(cpu|memory|disk|connection)"
psql -h heliosdb-lb -U admin -c "SELECT datname, numbackends, blks_read, blks_hit FROM pg_stat_database;"

Capacity Planning

Plan capacity from:

  • Peak query concurrency.
  • Write throughput and WAL growth.
  • Data retention and index growth.
  • Backup and restore windows.
  • Regional failover requirements.

Keep enough headroom for one node or one region to fail without breaching service objectives.

Cost Optimization

Reduce cost only after confirming sustained overprovisioning:

  1. Identify nodes with low utilization over a representative period.
  2. Confirm no failover or maintenance workload depends on that headroom.
  3. Scale down gradually.
  4. Monitor latency, error rate, and replication lag after each change.