Vector Search

Zatabase provides first-class vector search powered by ZQL, Zatabase’s patent-pending query engine. ZQL uses a novel approach to similarity search that delivers exceptional speed and memory efficiency. Zatabase also supports dense f32 vectors with HNSW approximate nearest neighbor indexing and KNN query syntax through both SQL and REST APIs.

Vector Columns

Define vector columns using the VECTOR(N) type, where N is the vector dimension:

CREATE TABLE embeddings (
  title TEXT,
  content TEXT,
  embedding VECTOR(384)
);

Insert vectors using the ARRAY[...] literal syntax:

INSERT INTO embeddings (title, content, embedding)
VALUES (
  'Machine Learning Basics',
  'An introduction to ML concepts...',
  ARRAY[0.12, -0.34, 0.56, ..., 0.78]
);

Distance Metrics

Zatabase supports three distance metrics for vector similarity:

Metric	Description	Use Case
L2 (Euclidean)	Straight-line distance between points	General-purpose, unnormalized vectors
Cosine	Angle between vectors (1 - cosine similarity)	Text embeddings, normalized representations
Inner Product	Dot product (negated for distance)	Maximum inner product search, recommendation

Dense f32 distance computations are optimized for throughput. Zatabase selects the best available implementation at runtime for L2, cosine, and inner product metrics.

KNN Search (Exact)

For exact K-nearest neighbor search, use the SQL distance operator with ORDER BY and LIMIT:

curl -s -X POST https://your-project.zatabase.io/v1/sql \
  -H "Authorization: Bearer $ZATABASE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM embeddings ORDER BY embedding <-> ARRAY[0.1, 0.2, ...] LIMIT 10"
  }'

Exact KNN computes the distance from the query vector to every row in the table. This guarantees perfect recall but is O(n) in the number of rows.

You can also use the dedicated KNN endpoint for batch operations:

curl -s -X POST https://your-project.zatabase.io/v1/knn \
  -H "Authorization: Bearer $ZATABASE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "table": "embeddings",
    "column": "embedding",
    "query": [0.1, 0.2, 0.3, ...],
    "k": 10,
    "metric": "cosine"
  }'

HNSW Index (Approximate)

For large datasets, build an HNSW index to enable sub-linear approximate nearest neighbor (ANN) search.

Building an Index

curl -s -X POST https://your-project.zatabase.io/v1/indexes/hnsw/build \
  -H "Authorization: Bearer $ZATABASE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "table": "embeddings",
    "column": "embedding",
    "metric": "cosine",
    "m": 16,
    "ef_construction": 200
  }'

HNSW Parameters

Parameter	Default	Description
m	16	Maximum number of connections per node per layer. Higher values improve recall but increase memory and build time. Typical range: 12-48.
ef_construction	200	Size of the dynamic candidate list during index construction. Higher values improve index quality at the cost of slower builds. Typical range: 100-500.
metric	"l2"	Distance metric: "l2", "cosine", or "inner_product". Must match your data characteristics.

Tuning guidelines:

• Small datasets (< 10K vectors): m=12, ef_construction=100 — fast builds, good recall
• Medium datasets (10K-1M): m=16, ef_construction=200 — balanced performance
• Large datasets (> 1M): m=32, ef_construction=400 — higher recall, slower builds

Searching with an Index

curl -s -X POST https://your-project.zatabase.io/v1/indexes/hnsw/search \
  -H "Authorization: Bearer $ZATABASE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "table": "embeddings",
    "column": "embedding",
    "query": [0.1, 0.2, 0.3, ...],
    "k": 10,
    "ef_search": 100
  }'

Tuning ef_search for Recall vs Speed

The ef_search parameter controls the tradeoff between search accuracy and latency at query time:

ef_search	Recall	Latency	Use Case
10	~85%	Fastest	Real-time suggestions, low-stakes retrieval
50	~95%	Fast	Production search, recommendation engines
100	~98%	Moderate	High-quality search (default)
200	~99.5%	Slower	Near-exact results, evaluation benchmarks
400+	~100%	Slowest	When recall matters more than speed

ef_search must be >= k (the number of results requested). Setting ef_search = k gives the fastest but lowest-quality results.

Index Management

List all HNSW indexes:

curl -s https://your-project.zatabase.io/v1/indexes/hnsw \
  -H "Authorization: Bearer $ZATABASE_TOKEN" | jq

Get index statistics:

curl -s https://your-project.zatabase.io/v1/indexes/hnsw/embeddings/embedding/stats \
  -H "Authorization: Bearer $ZATABASE_TOKEN" | jq

ZQL Native Similarity Search

In addition to dense f32 vectors, Zatabase’s ZQL engine provides a patent-pending native similarity search that delivers ultra-fast comparisons with minimal memory overhead.

ZQL similarity search is accessed via the ZQL API:

# Similarity search using the ~ operator
curl -s -X POST https://your-project.zatabase.io/api/zql/search \
  -H "Authorization: Bearer $ZATABASE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "store": "documents",
    "field": "content",
    "value": "machine learning introduction",
    "threshold": 0.6,
    "limit": 10
  }'

The threshold parameter (0.0 to 1.0) filters results by minimum similarity. Meaningful matches typically have similarity > 0.55.

Performance Notes

• ZQL native search: ZQL’s patent-pending representation enables extremely fast similarity search with minimal memory overhead
• Dense f32 vectors: Standard floating-point vectors support L2, cosine, and inner product metrics with optimized distance computations
• Batch processing: The HNSW search endpoint processes distance calculations in optimized batches with top-k selection
• Memory efficiency: HNSW graphs are stored on disk and memory-mapped for large-scale deployments
• O(1) exact match: The = operator provides instant exact matching, bypassing similarity search entirely