> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cyborg.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure an Encrypted Index

CyborgDB offers three index types, all of which offer varying characteristics:

|            Index Type            |  Speed  |  Recall | Index Size |
| :------------------------------: | :-----: | :-----: | :--------: |
| [`IVFFlat`](#ivfflat-index-type) |   Fast  | Highest |   Biggest  |
|   [`IVFPQ`](#ivfpq-index-type)   |   Fast  |   High  |   Medium   |
|     [`IVF`](#ivf-index-type)     | Fastest |  Lowest |  Smallest  |

Generally-speaking, we recommend that you start with `IVFFlat`, which provides the highest recall (accuracy) with high indexing and retrieval speeds. For most use-cases, this index type will scale well. If minimizing index size is important, `IVFPQ` takes `IVFFlat` and applies Product-Quantization (PQ, a form of lossy compression) to the vector embeddings which can greatly reduce index size.

## `IVFFlat` Index Type

The `IVFFlat` index type improves `IVF` significantly by storing encrypted vector embeddings in the index. In addition to selecting the closest clusters for a query vector, the exact distance can be computed between each candidate vector and the query, yielding very high recall rates (up to >99%). This comes at the cost of index size and some search speed.

We recommend `IVFFlat` indexes for most applications, as it provides the highest recall rates, and it's possible to mitigate index size constraints later via `IVFPQ`.

To create an `IVFFlat` index, you can use its configuration constructor:

<CodeGroup>
  ```python Python SDK icon="python" theme={null}
  from cyborgdb import Client, IndexIVFFlat, generate_key

  # Create a client
  client = Client('http://localhost:8000', 'your-api-key')

  # Set index parameters
  dimension = 128  # Adjust according to your dataset (dimensionality)
  n_lists = 4096

  # Create the IVFFlat index config
  index_config = IndexIVFFlat(
      type='ivf_flat',
      dimension=dimension,
      n_lists=n_lists
  )

  # Generate encryption key and create the index
  index_name = "test_index"
  index_key = generate_key()  # Generate secure 32-byte key
  index = client.create_index(index_name, index_key, index_config)
  ```

  ```javascript JavaScript SDK icon="js" theme={null}
  import { Client } from 'cyborgdb';

  // Create a client
  const client = new Client('http://localhost:8000', 'your-api-key');

  // Set index parameters
  const dimension = 128;  // Adjust according to your dataset (dimensionality)
  const nLists = 4096;

  // Create the IVFFlat index config
  const indexConfig = {
      type: 'ivf_flat',
      dimension: dimension,
      nLists: nLists
  };

  // Generate encryption key and create the index
  const indexName = "test_index";
  const indexKey = crypto.getRandomValues(new Uint8Array(32)); // Generate secure 32-byte key
  const index = await client.createIndex(indexName, indexKey, indexConfig);
  ```

  ```typescript TypeScript SDK icon="code" theme={null}
  import { Client, IndexIVFFlatModel } from 'cyborgdb';

  // Create a client
  const client = new Client('http://localhost:8000', 'your-api-key');

  // Set index parameters
  const dimension = 128;  // Adjust according to your dataset (dimensionality)
  const nLists = 4096;

  // Create the IVFFlat index config
  const indexConfig: IndexIVFFlatModel = {
      type: 'ivf_flat',
      dimension: dimension,
      nLists: nLists
  };

  // Generate encryption key and create the index
  const indexName = "test_index";
  const indexKey = crypto.getRandomValues(new Uint8Array(32)); // Generate secure 32-byte key
  const index = await client.createIndex(indexName, indexKey, indexConfig);
  ```
</CodeGroup>

`n_lists` is the number of clusters into which each vector in the index can be categorized. Typically, the higher the value, the higher the recall (but also the slower the indexing process). As a good rule of thumbs, `n_lists` should be:

* A base-2 number (e.g., `2,048`, `4,096`). Not a requirement, but yields performance optimizations.
* Each cluster should have between `100` - `10,000` vectors; so `n_lists` should be roughly between `1/100` - `1/10,000` of the total number of items which will be indexed.

## `IVFPQ` Index Type

<Note>The `IVFPQ` index type is only supported in paid plans.</Note>

The `IVFPQ` index type builds upon `IVFFlat` by applying Product Quantization (PQ) - a form of lossy compression - to reduce the index size. When applied correctly, `IVFPQ` indexes can maintain high recall (>95%) while reducing index size significantly (2-4x).

We recommend `IVFPQ` indexes for mature applications, where the dataset and query distributions are well-established. This is because `IVFPQ` requires the most tuning to yield an ideal balance between recall and index size. It is possible to go from `IVFFlat` to `IVFPQ` on the same index, but not vice-versa.

To create an `IVFPQ` index, you can use its configuration constructor:

<CodeGroup>
  ```python Python SDK icon="python" theme={null}
  from cyborgdb import Client, IndexIVFPQ, generate_key

  # Create a client
  client = Client('http://localhost:8000', 'your-api-key')

  # Set index parameters
  dimension = 128  # Adjust according to your dataset (dimensionality)
  n_lists = 4096
  pq_dim = 32 # Dimension must be divisible by pq_dim
  pq_bits = 8 # Number of bits for each pq dimension

  # Create the IVFPQ index config
  index_config = IndexIVFPQ(
      type='ivf_pq',
      dimension=dimension,
      n_lists=n_lists,
      pq_dim=pq_dim,
      pq_bits=pq_bits
  )

  # Generate encryption key and create the index
  index_name = "test_index"
  index_key = generate_key()  # Generate secure 32-byte key
  index = client.create_index(index_name, index_key, index_config)
  ```

  ```javascript JavaScript SDK icon="js" theme={null}
  import { Client } from 'cyborgdb';

  // Create a client
  const client = new Client('http://localhost:8000', 'your-api-key');

  // Set index parameters
  const dimension = 128;  // Adjust according to your dataset (dimensionality)
  const nLists = 4096;
  const pqDim = 32; // Dimension must be divisible by pqDim
  const pqBits = 8; // Number of bits for each pq dimension

  // Create the IVFPQ index config
  const indexConfig = {
      type: 'ivf_pq',
      dimension: dimension,
      nLists: nLists,
      pqDim: pqDim,
      pqBits: pqBits
  };

  // Generate encryption key and create the index
  const indexName = "test_index";
  const indexKey = crypto.getRandomValues(new Uint8Array(32)); // Generate secure 32-byte key
  const index = await client.createIndex(indexName, indexKey, indexConfig);
  ```

  ```typescript TypeScript SDK icon="code" theme={null}
  import { Client, IndexIVFPQModel } from 'cyborgdb';

  // Create a client
  const client = new Client('http://localhost:8000', 'your-api-key');

  // Set index parameters
  const dimension = 128;  // Adjust according to your dataset (dimensionality)
  const nLists = 4096;
  const pqDim = 32; // Dimension must be divisible by pqDim
  const pqBits = 8; // Number of bits for each pq dimension

  // Create the IVFPQ index config
  const indexConfig: IndexIVFPQModel = {
      type: 'ivf_pq',
      dimension: dimension,
      nLists: nLists,
      pqDim: pqDim,
      pqBits: pqBits
  };

  // Generate encryption key and create the index
  const indexName = "test_index";
  const indexKey = crypto.getRandomValues(new Uint8Array(32)); // Generate secure 32-byte key
  const index = await client.createIndex(indexName, indexKey, indexConfig);
  ```
</CodeGroup>

`n_lists` is the number of clusters into which each vector in the index can be categorized. Typically, the higher the value, the higher the recall (but also the slower the indexing process). As a good rule of thumbs, `n_lists` should be:

* A base-2 number (e.g., `2,048`, `4,096`). Not a requirement, but yields performance optimizations.
* Each cluster should have between `100` - `10,000` vectors; so `n_lists` should be roughly between `1/100` - `1/10,000` of the total number of items which will be indexed.

`pq_dim` is the number of dimensionality for each vector after product-quantization. It must be between `1` and `dimension`, and `dimension` must be cleanly divisible by `pq_dim`. Lower `pq_dim` will yield smaller index sizes but lower recall.

`pq_bits` is the number of bits that will be used to represent each dimension of the product-quantized vector embeddings. It must be between `1` and `16`, with lower values yielding smaller index sizes but lower recall.

## `IVF` Index Type

<Note>The `IVF` index type is only supported in paid plans.</Note>

The `IVF` index type (Inverted File Index) is the simplest offered by CyborgDB. It takes a single parameter, `n_lists`, which defines the number of lists, or clusters, into which the index is segmented. In other words, vectors that are close together will be clustered together into a single list. At retrieval time, vectors in the clusters closest to the query vector will be returned. This yields a very fast, but rather inaccurate, search.

We recommend `IVF` indexes for applications which require high-speed, low-latency search with low recall requirements (or where `top_k` is rather large, i.e. >500).

To create an `IVF` index, you can use its configuration constructor:

<CodeGroup>
  ```python Python SDK icon="python" theme={null}
  from cyborgdb import Client, IndexIVF, generate_key

  # Create a client
  client = Client('http://localhost:8000', 'your-api-key')

  # Set index parameters
  dimension = 128  # Adjust according to your dataset (dimensionality)
  n_lists = 4096

  # Create the IVF index config
  index_config = IndexIVF(
      type='ivf',
      dimension=dimension,
      n_lists=n_lists
  )

  # Generate encryption key and create the index
  index_name = "test_index"
  index_key = generate_key()  # Generate secure 32-byte key
  index = client.create_index(index_name, index_key, index_config)
  ```

  ```javascript JavaScript SDK icon="js" theme={null}
  import { Client } from 'cyborgdb';

  // Create a client
  const client = new Client('http://localhost:8000', 'your-api-key');

  // Set index parameters
  const dimension = 128;  // Adjust according to your dataset (dimensionality)
  const nLists = 4096;

  // Create the IVF index config
  const indexConfig = {
      type: 'ivf',
      dimension: dimension,
      nLists: nLists
  };

  // Generate encryption key and create the index
  const indexName = "test_index";
  const indexKey = crypto.getRandomValues(new Uint8Array(32)); // Generate secure 32-byte key
  const index = await client.createIndex(indexName, indexKey, indexConfig);
  ```

  ```typescript TypeScript SDK icon="code" theme={null}
  import { Client, IndexIVFModel } from 'cyborgdb';

  // Create a client
  const client = new Client('http://localhost:8000', 'your-api-key');

  // Set index parameters
  const dimension = 128;  // Adjust according to your dataset (dimensionality)
  const nLists = 4096;

  // Create the IVF index config
  const indexConfig: IndexIVFModel = {
      type: 'ivf',
      dimension: dimension,
      nLists: nLists
  };

  // Generate encryption key and create the index
  const indexName = "test_index";
  const indexKey = crypto.getRandomValues(new Uint8Array(32)); // Generate secure 32-byte key
  const index = await client.createIndex(indexName, indexKey, indexConfig);
  ```
</CodeGroup>

`n_lists` is the number of clusters into which each vector in the index can be categorized. Typically, the higher the value, the higher the recall (but also the slower the indexing process). As a good rule of thumbs, `n_lists` should be:

* A base-2 number (e.g., `2,048`, `4,096`). Not a requirement, but yields performance optimizations.
* Each cluster should have between `100` - `10,000` vectors; so `n_lists` should be roughly between `1/100` - `1/10,000` of the total number of items which will be indexed.

## Customizing Distance Metrics

By default, CyborgDB uses `euclidean` distance as its metric for all index types. You can override this default by providing a `metric` parameter to any of the index constructors. For example:

<CodeGroup>
  ```python Python SDK icon="python" theme={null}
  # Example with cosine similarity
  index_config = IndexIVFFlat(
      type='ivf_flat',
      dimension=dimension,
      n_lists=n_lists,
      metric='cosine'
  )
  ```

  ```javascript JavaScript SDK icon="js" theme={null}
  // Example with cosine similarity
  const indexConfig = {
      type: 'ivf_flat',
      dimension: dimension,
      nLists: nLists,
      metric: 'cosine'
  };
  ```

  ```typescript TypeScript SDK icon="code" theme={null}
  // Example with cosine similarity
  const indexConfig: IndexIVFFlatModel = {
      type: 'ivf_flat',
      dimension: dimension,
      nLists: nLists,
      metric: 'cosine'
  };
  ```

  ```bash cURL icon="rectangle-terminal" theme={null}
  # Example with cosine similarity
  curl -X POST "http://localhost:8000/v1/indexes/create" \
       -H "X-API-Key: your-api-key" \
       -H "Content-Type: application/json" \
       -d '{
         "index_name": "my_index",
         "index_key": "your_64_character_hex_key_here",
         "index_config": {
           "type": "ivfflat",
           "dimension": 768,
           "n_lists": 1024,
           "metric": "cosine"
         }
       }'
  ```
</CodeGroup>

The currently supported distance metrics are:

* `"cosine"`: Cosine similarity.
* `"euclidean"`: Euclidean distance.
* `"squared_euclidean"`: Squared Euclidean distance.

## API Reference

For more information on the `IndexConfig` classes, refer to the API Reference:

<CardGroup cols={3}>
  <Card title="REST API Reference" href="../../rest-api/client/create-index#index-types" icon="rectangle-terminal">
    REST API reference for `/v1/indexes/create`
  </Card>

  <Card title="Python SDK Reference" href="../../python-sdk/types#indexconfig" icon="python">
    API reference for `IndexConfig` in Python
  </Card>

  <Card title="JS/TS SDK Reference" href="../../js-ts-sdk/types#indexconfig" icon="js">
    API reference for `IndexConfig` in JavaScript/TypeScript
  </Card>
</CardGroup>