Train an Encrypted Index

CyborgDB Service stores every encrypted index as an inverted-file (IVF) index whose clusters are produced by a training step. Training fits the clustering model to the actual vectors in your index so queries can probe a small number of clusters instead of scanning everything. In CyborgDB Service, training is handled automatically: by default, an index is trained once it holds more than 65,536 vectors, and retrained as it grows (see How automatic training works). You can also trigger training explicitly once enough vectors have been added, for example to specify training parameters.

# Train the encrypted index
index.train()

# Or train with a specific number of clusters
index.train(n_lists=128)

// Train the encrypted index
await index.train();

// Or train with a specific number of clusters
await index.train({ nLists: 128 });

// Train the encrypted index
await index.train();

// Or train with a specific number of clusters
await index.train({ nLists: 128 });

// Train the encrypted index
err := index.Train(context.Background(), cyborgdb.TrainParams{})

// Or train with a specific number of clusters
nLists := int32(128)
err = index.Train(context.Background(), cyborgdb.TrainParams{
    NLists: &nLists,
})

curl -X POST "http://localhost:8000/v1/indexes/train" \
     -H "X-API-Key: your-api-key" \
     -H "Content-Type: application/json" \
     -d '{
       "index_name": "my_index",
       "index_key": "your_64_character_hex_key_here"
     }'

Training requires at least 10,000 vectors and at least 2 * n_lists vectors in the index (ingested via upsert). If the index has fewer, train does not return an error: the service logs a warning and the index stays untrained. Use is_trained() to check whether training took effect.

Training Parameters

Parameters are available to customize the training process:

Parameter	Type	Default	Description
`n_lists`	`int`	`None` (auto)	(Optional) Number of inverted index lists to create in the index. When `None` or omitted, auto-determines based on the number of vectors in the index.
`batch_size`	`int`	`None`	(Optional) Number of vectors to process per training batch. When `None`, the server uses 2048.
`max_iters`	`int`	`None`	(Optional) Maximum number of training iterations. When `None`, the server uses 100.
`tolerance`	`float`	`None`	(Optional) Convergence tolerance for training completion. When `None`, the server uses 1e-6.
`max_memory`	`int`	`None` (0)	(Optional) Maximum memory usage in MB. When `None` or 0, no memory limit is applied.

n_lists is the number of clusters into which each vector in the index can be categorized. Typically, the higher the value, the higher the recall (but also the slower the indexing process). As a good rule of thumb, n_lists should be:

A base-2 number (e.g., 2,048, 4,096). Not a requirement, but yields performance optimizations.
Each cluster should have between 100 - 10,000 vectors; so n_lists should be roughly between 1/100 - 1/10,000 of the total number of items which will be indexed.

If not specified, CyborgDB will auto-determine the best n_lists value based on the number of vectors in the index.

How automatic training works

After each upsert, the service checks whether the index should be trained or retrained:

First training. An untrained index is trained automatically once it holds more than AUTO_TRAIN_MIN_VECTORS vectors (default 65536).
Retraining. A trained index is retrained automatically when the optimal n_lists for its current vector count grows past the index’s current n_lists. The optimal n_lists is the square root of the vector count, rounded to a power of two and capped at 65,536. In practice, an index is retrained roughly each time its vector count grows about 4x.

Automatic training uses the optimal n_lists for the vector count at the time it runs. Training runs in the background, so the upsert that triggers it returns immediately. For an index that was first trained automatically, the progression looks like this:

Vectors in index	Optimal `n_lists`	Automatic action
65,536 or fewer	—	None (index stays untrained)
65,537	256	First training
131,072	512	Retrain
524,288	1,024	Retrain
2,097,152	2,048	Retrain
8,388,608	4,096	Retrain
33,554,432	8,192	Retrain

If you train with an explicit n_lists, the index is retrained automatically only once the optimal n_lists exceeds the value you chose. To change the first-training threshold, set AUTO_TRAIN_MIN_VECTORS. To turn off automatic training and retraining entirely, set AUTO_TRAIN_DISABLED=true; you then train indexes yourself with train(). See the Environment Variables guide.

Avoid the large-untrained-query warning

While training is technically optional (you can use CyborgDB without ever calling train), it is recommended that you do so once you have a large number of vectors in the index (e.g., > 50,000). If you don’t, and you call query, you will see a warning in the console, stating:

Warning: querying untrained index with more than 50000 indexed vectors.

API Reference

For more information on training an encrypted index, refer to the API reference:

REST API Reference

REST API reference for /v1/indexes/train

Python SDK Reference

API reference for train() in Python

JS/TS SDK Reference

API reference for train() in JavaScript/TypeScript

Go SDK Reference

API reference for Train() in Go

CyborgDB Service

Encrypted Indexes

Data Operations

Advanced

Train an Encrypted Index

Training Parameters

How automatic training works

Avoid the large-untrained-query warning

API Reference

REST API Reference

Python SDK Reference

JS/TS SDK Reference

Go SDK Reference

​Training Parameters

​How automatic training works

​Avoid the large-untrained-query warning

​API Reference

REST API Reference

Python SDK Reference

JS/TS SDK Reference

Go SDK Reference

Training Parameters

How automatic training works

Avoid the large-untrained-query warning

API Reference