Skip to content
HeliosDB HeliosDB Nano

Product Quantization Integration Guide

Product Quantization Integration Guide

Overview

Product Quantization (PQ) is now fully integrated with HeliosDB Nano’s storage layer, enabling memory-efficient vector indexing with 8-16x compression while maintaining 95-98% search accuracy.

Features

  • SQL Support: Create PQ indexes using standard SQL syntax
  • Automatic Configuration: Smart defaults based on vector dimensions
  • Persistence: Codebooks and quantized vectors are fully persisted
  • Transparency: Works seamlessly with existing vector operations

SQL Syntax

Basic PQ Index Creation

-- Create table with vector column
CREATE TABLE documents (
    id INTEGER PRIMARY KEY,
    title TEXT,
    embedding VECTOR(768)
);


-- Create PQ-compressed vector index
CREATE INDEX doc_embedding_idx ON documents(embedding)
USING hnsw
WITH (quantization='product');

Advanced Configuration

-- Customize PQ parameters
CREATE INDEX custom_pq_idx ON documents(embedding)
USING hnsw
WITH (
    quantization='product',
    pq_subquantizers=8,      -- Number of sub-quantizers (M)
    pq_centroids=256          -- Centroids per sub-quantizer (K)
);

Parameter Guidelines

pq_subquantizers (M):

  • Default: Auto-calculated based on dimension
    • 768+ dims: M=8
    • 256-767 dims: M=4
    • <256 dims: M=2
  • Must evenly divide vector dimension
  • Higher M = better accuracy, larger codes
  • Typical range: 2-16

pq_centroids (K):

  • Default: 256 (recommended)
  • Range: 1-256
  • 256 allows 1-byte codes (optimal)
  • Higher K = better accuracy, larger codebook

Usage Examples

1. Create and Populate Index

-- Create table
CREATE TABLE embeddings (
    id INTEGER PRIMARY KEY,
    text TEXT,
    vector VECTOR(384)
);


-- Create PQ index
CREATE INDEX emb_idx ON embeddings(vector)
USING hnsw
WITH (quantization='product');


-- Insert vectors (automatically quantized)
INSERT INTO embeddings (id, text, vector) VALUES
(1, 'hello world', VECTOR([0.1, 0.2, ...])),
(2, 'foo bar', VECTOR([0.3, 0.4, ...])),
(3, 'test data', VECTOR([0.5, 0.6, ...]));

2. Search with PQ Index

-- Vector similarity search (uses PQ automatically)
SELECT id, text, vector_distance(vector, VECTOR([0.1, 0.2, ...])) as distance
FROM embeddings
ORDER BY distance
LIMIT 10;

3. Check Index Statistics

-- View index details
SELECT
    index_name,
    num_vectors,
    dimensions,
    quantization,
    memory_bytes,
    memory_bytes / 1024.0 / 1024.0 as memory_mb
FROM helios_vector_indexes
WHERE quantization = 'Product';

4. Compare Memory Usage

-- Create standard and PQ indexes for comparison
CREATE TABLE comparison (
    id INTEGER PRIMARY KEY,
    vec VECTOR(768)
);


CREATE INDEX std_idx ON comparison(vec) USING hnsw;
CREATE INDEX pq_idx ON comparison(vec) USING hnsw
WITH (quantization='product');


-- Insert test data
INSERT INTO comparison (id, vec)
SELECT
    generate_series(1, 10000),
    random_vector(768);


-- Compare memory usage
SELECT
    index_name,
    quantization,
    memory_bytes,
    memory_bytes / 1024.0 / 1024.0 as memory_mb
FROM helios_vector_indexes
WHERE table_name = 'comparison';

Expected output:

index_name | quantization | memory_bytes | memory_mb
-----------+--------------+--------------+----------
std_idx    | None         | 30720000     | 29.30
pq_idx     | Product      | 2048000      | 1.95

Configuration Examples by Dimension

Common Embedding Dimensions

384 dimensions (MiniLM, etc.):

CREATE INDEX idx_384 ON table(vec) USING hnsw
WITH (
    quantization='product',
    pq_subquantizers=4,    -- 384/4 = 96 per subquantizer
    pq_centroids=256
);

768 dimensions (BERT, RoBERTa):

CREATE INDEX idx_768 ON table(vec) USING hnsw
WITH (
    quantization='product',
    pq_subquantizers=8,    -- 768/8 = 96 per subquantizer
    pq_centroids=256
);

1536 dimensions (OpenAI ada-002):

CREATE INDEX idx_1536 ON table(vec) USING hnsw
WITH (
    quantization='product',
    pq_subquantizers=8,    -- 1536/8 = 192 per subquantizer
    pq_centroids=256
);

Performance Characteristics

Memory Compression

DimensionStandardPQ (M=8)Compression
128512 B40 B12.8x
3841.5 KB40 B38.4x
7683.0 KB40 B76.8x
15366.0 KB40 B153.6x

Note: Actual compression includes codebook overhead

Search Accuracy

Typical recall@10 metrics:

  • 95-98% with M=8, K=256
  • 92-95% with M=4, K=256
  • 88-92% with M=2, K=256

Query Performance

  • PQ Search: ~5-10% slower than standard HNSW
  • ADC Optimization: Precomputed distance tables minimize overhead
  • Large Datasets: Performance gap decreases as dataset grows

Best Practices

1. Training Data

-- Let the index train on existing data
CREATE INDEX pq_idx ON large_table(vec) USING hnsw
WITH (quantization='product');


-- Index will use sample of existing vectors for training
-- Minimum 10,000 vectors recommended for optimal training

2. Dimension Selection

  • Choose M such that D/M is between 64-128
  • Common configurations:
    • 384 dims: M=4 (96 per subquantizer)
    • 768 dims: M=8 (96 per subquantizer)
    • 1536 dims: M=12 or M=16

3. When to Use PQ

Use PQ when:

  • Dataset > 100K vectors
  • Memory is constrained
  • High-dimensional vectors (>256 dims)
  • Can tolerate 2-5% accuracy loss

Use standard HNSW when:

  • Dataset < 10K vectors
  • Maximum accuracy required
  • Low-dimensional vectors (<128 dims)
  • Memory is not constrained

Troubleshooting

Error: Dimension not divisible by subquantizers

-- This will fail: 100 is not divisible by 8
CREATE INDEX bad_idx ON table(vec VECTOR(100)) USING hnsw
WITH (quantization='product', pq_subquantizers=8);


-- Solution: Use M=4 (100/4 = 25)
CREATE INDEX good_idx ON table(vec VECTOR(100)) USING hnsw
WITH (quantization='product', pq_subquantizers=4);

Insufficient Training Data

-- Error if table has < 10,000 vectors
CREATE INDEX pq_idx ON small_table(vec) USING hnsw
WITH (quantization='product');


-- Solution: Use standard index for small datasets
CREATE INDEX std_idx ON small_table(vec) USING hnsw;

Low Search Accuracy

-- If recall is too low, increase M
CREATE INDEX better_pq_idx ON table(vec VECTOR(768)) USING hnsw
WITH (
    quantization='product',
    pq_subquantizers=16  -- Increased from 8
);

Migration Guide

Migrating Existing Indexes to PQ

-- 1. Create new PQ index
CREATE INDEX new_pq_idx ON documents(embedding) USING hnsw
WITH (quantization='product');


-- 2. Test search quality
SELECT id FROM documents
USE INDEX (new_pq_idx)
ORDER BY vector_distance(embedding, VECTOR(?))
LIMIT 10;


-- 3. Drop old index if satisfied
DROP INDEX old_idx;

Monitoring

Index Health

SELECT
    index_name,
    num_vectors,
    quantization,
    memory_bytes,
    memory_bytes::float / num_vectors as bytes_per_vector,
    recall_at_10
FROM helios_vector_indexes
WHERE quantization = 'Product';

Performance Metrics

-- Track query performance
EXPLAIN ANALYZE
SELECT id FROM documents
ORDER BY vector_distance(embedding, VECTOR(?))
LIMIT 10;

Advanced Topics

Custom Distance Metrics

-- PQ supports different distance metrics
CREATE INDEX cosine_pq_idx ON documents(embedding) USING hnsw
WITH (
    quantization='product',
    distance_metric='cosine'
);

Hybrid Indexes

-- Use PQ for large-scale search, standard for refinement
CREATE INDEX pq_idx ON documents(embedding) USING hnsw
WITH (quantization='product');


CREATE INDEX std_idx ON top_results(embedding) USING hnsw;


-- Two-stage search
WITH candidates AS (
    SELECT id FROM documents
    USE INDEX (pq_idx)
    ORDER BY vector_distance(embedding, VECTOR(?))
    LIMIT 100
)
SELECT id FROM candidates
JOIN documents USING (id)
USE INDEX (std_idx)
ORDER BY vector_distance(embedding, VECTOR(?))
LIMIT 10;

References