Configure an Encrypted Index

Index configuration is automatically handled by default. This guide allows you to override these defaults to customize index behavior & performance characteristics.

CyborgDB offers three index types, all of which offer varying characteristics:

Index Type	Speed	Accuracy	Memory Usage
`IVFFlat`	Medium	Highest	High
`IVFPQ`	Fast	Good	Low
`IVFSQ`	Fast	High	Low

Generally-speaking, we recommend that you start with IVFFlat, which provides the highest recall (accuracy) with high indexing and retrieval speeds. For most use-cases, this index type will scale well. If minimizing index size is important, IVFPQ takes IVFFlat and applies Product-Quantization (PQ, a form of lossy compression) to the vector embeddings which can greatly reduce index size. For an alternative that balances speed and index size, IVFSQ uses Scalar Quantization to compress each dimension independently.

`IVFFlat` Index Type

The IVFFlat index type improves IVF significantly by storing encrypted vector embeddings in the index. In addition to selecting the closest clusters for a query vector, the exact distance can be computed between each candidate vector and the query, yielding very high recall rates (up to >99%). This comes at the cost of index size and some search speed. We recommend IVFFlat indexes for most applications, as it provides the highest recall rates, and it’s possible to mitigate index size constraints later via IVFPQ. To create an IVFFlat index, you can use its configuration constructor:

from cyborgdb import Client, IndexIVFFlat

# Create a client
client = Client(
    base_url='http://localhost:8000', 
    api_key='your-api-key'
)

# Create the IVFFlat index config
index_config = IndexIVFFlat()

# Generate encryption key and create the index
index_name = "test_index"
index_key = client.generate_key(save=True)  # Generate secure 32-byte key
index = client.create_index(
    index_name=index_name, 
    index_key=index_key, 
    index_config=index_config
)

import { Client } from 'cyborgdb';

// Create a client
const client = new Client({
    baseUrl: 'http://localhost:8000',
    apiKey: 'your-api-key'
});

// Create the IVFFlat index config
const indexConfig = { type: 'ivfflat' };

// Generate encryption key and create the index
const indexName = "test_index";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig
});

import { Client, IndexIVFFlat } from 'cyborgdb';

// Create a client
const client = new Client({
    baseUrl: 'http://localhost:8000',
    apiKey: 'your-api-key'
});

// Create the IVFFlat index config
const indexConfig: IndexIVFFlat = { type: 'ivfflat' };

// Generate encryption key and create the index
const indexName = "test_index";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig
});

package main

import (
    "context"
    "log"

    "github.com/cyborginc/cyborgdb-go"
)

func main() {
    // Create a client
    client, err := cyborgdb.NewClient("http://localhost:8000", "your-api-key")
    if err != nil {
        log.Fatal(err)
    }

    // Create the IVFFlat index config (specify dimension, e.g., 768 for many embedding models)
    indexConfig := cyborgdb.IndexIVFFlat(768)

    // Generate encryption key and create the index
    indexName := "test_index"
    indexKey, err := cyborgdb.GenerateKey()
    if err != nil {
        log.Fatal(err)
    }

    index, err := client.CreateIndex(context.Background(), &cyborgdb.CreateIndexParams{
        IndexName:   indexName,
        IndexKey:    indexKey,
        IndexConfig: indexConfig,
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Created index: %s", index.GetIndexName())
}

`IVFPQ` Index Type

The IVFPQ index type builds upon IVFFlat by applying Product Quantization (PQ) - a form of lossy compression - to reduce the index size. When applied correctly, IVFPQ indexes can maintain high recall (>95%) while reducing index size significantly (2-4x). We recommend IVFPQ indexes for mature applications, where the dataset and query distributions are well-established. This is because IVFPQ requires the most tuning to yield an ideal balance between recall and index size. It is possible to go from IVFFlat to IVFPQ on the same index, but not vice-versa. To create an IVFPQ index, you can use its configuration constructor:

from cyborgdb import Client, IndexIVFPQ

# Create a client
client = Client(
    base_url='http://localhost:8000', 
    api_key='your-api-key'
)

# Set index parameters
pq_dim = 32 # Dimension must be divisible by pq_dim
pq_bits = 8 # Number of bits for each pq dimension

# Create the IVFPQ index config
index_config = IndexIVFPQ(pq_dim, pq_bits)

# Generate encryption key and create the index
index_name = "test_index"
index_key = client.generate_key(save=True)  # Generate secure 32-byte key

index = client.create_index(
    index_name=index_name, 
    index_key=index_key, 
    index_config=index_config
)

import { Client } from 'cyborgdb';

// Create a client
const client = new Client({
    baseUrl: 'http://localhost:8000',
    apiKey: 'your-api-key'
});

// Set index parameters
const pqDim = 32; // Dimension must be divisible by pqDim
const pqBits = 8; // Number of bits for each pq dimension

// Create the IVFPQ index config
const indexConfig = { type: 'ivfpq', pqDim, pqBits };

// Generate encryption key and create the index
const indexName = "test_index";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig
});

import { Client, IndexIVFPQ } from 'cyborgdb';

// Create a client
const client = new Client({
    baseUrl: 'http://localhost:8000',
    apiKey: 'your-api-key'
});

// Set index parameters
const pqDim = 32; // Dimension must be divisible by pqDim
const pqBits = 8; // Number of bits for each pq dimension

// Create the IVFPQ index config
const indexConfig: IndexIVFPQ = { type: 'ivfpq', pqDim, pqBits };

// Generate encryption key and create the index
const indexName = "test_index";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig
});

package main

import (
    "context"
    "log"

    "github.com/cyborginc/cyborgdb-go"
)

func main() {
    // Create a client
    client, err := cyborgdb.NewClient("http://localhost:8000", "your-api-key")
    if err != nil {
        log.Fatal(err)
    }

    // Set index parameters
    dimension := int32(768)  // Vector dimension
    pqDim := int32(32)       // Dimension must be divisible by pqDim
    pqBits := int32(8)       // Number of bits for each pq dimension

    // Create the IVFPQ index config
    indexConfig := cyborgdb.IndexIVFPQ(dimension, pqDim, pqBits)

    // Generate encryption key and create the index
    indexName := "test_index"
    indexKey, err := cyborgdb.GenerateKey()
    if err != nil {
        log.Fatal(err)
    }

    index, err := client.CreateIndex(context.Background(), &cyborgdb.CreateIndexParams{
        IndexName:   indexName,
        IndexKey:    indexKey,
        IndexConfig: indexConfig,
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Created index: %s", index.GetIndexName())
}

pq_dim is the number of dimensionality for each vector after product-quantization. It must be between 1 and dimension, and dimension must be cleanly divisible by pq_dim. Lower pq_dim will yield smaller index sizes but lower recall. pq_bits is the number of bits that will be used to represent each dimension of the product-quantized vector embeddings. It must be between 1 and 16, with lower values yielding smaller index sizes but lower recall.

`IVFSQ` Index Type

The IVFSQ index type uses Scalar Quantization (SQ) to compress each vector dimension independently, balancing speed and index size. This makes it a good choice when you want fast search with a smaller index footprint while maintaining high recall. To create an IVFSQ index, you can use its configuration constructor:

from cyborgdb import Client, IndexIVFSQ

# Create a client
client = Client(
    base_url='http://localhost:8000',
    api_key='your-api-key'
)

# Create the IVFSQ index config
index_config = IndexIVFSQ(sq_bits=16)

# Generate encryption key and create the index
index_name = "test_index"
index_key = client.generate_key(save=True)  # Generate secure 32-byte key

index = client.create_index(
    index_name=index_name,
    index_key=index_key,
    index_config=index_config
)

import { Client } from 'cyborgdb';

// Create a client
const client = new Client({
    baseUrl: 'http://localhost:8000',
    apiKey: 'your-api-key'
});

// Create the IVFSQ index config
const indexConfig = { type: 'ivfsq', sqBits: 16 };

// Generate encryption key and create the index
const indexName = "test_index";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig
});

import { Client, IndexIVFSQ } from 'cyborgdb';

// Create a client
const client = new Client({
    baseUrl: 'http://localhost:8000',
    apiKey: 'your-api-key'
});

// Create the IVFSQ index config
const indexConfig: IndexIVFSQ = { type: 'ivfsq', sqBits: 16 };

// Generate encryption key and create the index
const indexName = "test_index";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig
});

package main

import (
    "context"
    "log"

    "github.com/cyborginc/cyborgdb-go"
)

func main() {
    // Create a client
    client, err := cyborgdb.NewClient("http://localhost:8000", "your-api-key")
    if err != nil {
        log.Fatal(err)
    }

    // Create the IVFSQ index config (specify dimension and sq_bits)
    indexConfig := cyborgdb.IndexIVFSQ(768, 16)

    // Generate encryption key and create the index
    indexName := "test_index"
    indexKey, err := cyborgdb.GenerateKey()
    if err != nil {
        log.Fatal(err)
    }

    index, err := client.CreateIndex(context.Background(), &cyborgdb.CreateIndexParams{
        IndexName:   indexName,
        IndexKey:    indexKey,
        IndexConfig: indexConfig,
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Created index: %s", index.GetIndexName())
}

sq_bits is the number of bits per dimension used for scalar quantization. Lower values yield smaller index sizes but may reduce recall. Accepted values are 8 and 16, with 16 being the default.

Customizing Distance Metrics

By default, CyborgDB uses euclidean distance as its metric for all index types. You can override this default by providing a metric parameter to any of the index constructors. For example:

# Example with cosine similarity
index_config = IndexIVFFlat()

index = client.create_index(
    index_name="index_name", 
    index_key=index_key, 
    index_config=index_config, 
    metric="cosine"
)

// Example with cosine similarity
const indexConfig = { type: 'ivfflat' };
const indexName = "index_name";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig,
    metric: "cosine"
});

// Example with cosine similarity
const indexConfig: IndexIVFFlat = { type: 'ivfflat' };
const indexName = "index_name";
const indexKey = client.generateKey(); // Generate secure 32-byte key

const index = await client.createIndex({
    indexName,
    indexKey,
    indexConfig,
    metric: "cosine"
});

package main

import (
    "context"
    "log"

    "github.com/cyborginc/cyborgdb-go"
)

func main() {
    // Create a client
    client, err := cyborgdb.NewClient("http://localhost:8000", "your-api-key")
    if err != nil {
        log.Fatal(err)
    }

    // Example with cosine similarity
    indexConfig := cyborgdb.IndexIVFFlat(768)
    indexName := "index_name"
    indexKey, err := cyborgdb.GenerateKey()
    if err != nil {
        log.Fatal(err)
    }

    metric := "cosine"
    index, err := client.CreateIndex(context.Background(), &cyborgdb.CreateIndexParams{
        IndexName:   indexName,
        IndexKey:    indexKey,
        IndexConfig: indexConfig,
        Metric:      &metric,
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Created index: %s", index.GetIndexName())
}

# Example with cosine similarity
curl -X POST "http://localhost:8000/v1/indexes/create" \
     -H "X-API-Key: your-api-key" \
     -H "Content-Type: application/json" \
     -d '{
       "index_name": "my_index",
       "index_key": "your_64_character_hex_key_here",
       "index_config": {
         "type": "ivfflat"
       },
       "metric": "cosine"
     }'

The currently supported distance metrics are:

"cosine": Cosine similarity.
"euclidean": Euclidean distance.
"squared_euclidean": Squared Euclidean distance.

API Reference

For more information on the IndexConfig classes, refer to the API Reference:

REST API Reference

REST API reference for /v1/indexes/create

Python SDK Reference

API reference for IndexConfig in Python

JS/TS SDK Reference

API reference for IndexConfig in JavaScript/TypeScript

Go SDK Reference

API reference for IndexConfig in Go

​IVFFlat Index Type

​IVFPQ Index Type

​IVFSQ Index Type

​Customizing Distance Metrics

​API Reference

REST API Reference

Python SDK Reference

JS/TS SDK Reference

Go SDK Reference

`IVFFlat` Index Type

`IVFPQ` Index Type

`IVFSQ` Index Type

Customizing Distance Metrics

API Reference