Create Index

Creates a new encrypted index on the service and returns an *EncryptedIndex bound to it.

func (c *Client) CreateIndex(ctx context.Context, params *CreateIndexParams) (*EncryptedIndex, error)

There is one index type, configured with flat fields. The earlier IndexIVFFlat, IndexIVFPQ, and IndexIVFSQ helpers and the IndexConfig field no longer exist.

Parameters

Parameter	Type	Description
`ctx`	`context.Context`	Context for cancellation and deadlines.
`params`	`*CreateIndexParams`	Index settings. A `nil` pointer returns a `*ValidationError` wrapping `ErrNilParams`.

CreateIndexParams

Optional scalar fields are pointers: leave them nil to use the service default, or set them with the cyborgdb.Int32, cyborgdb.String, cyborgdb.Float64, and cyborgdb.Bool helpers.

Field	Type	Default	Description
`IndexName`	`string`	-	Unique name for the index.
`IndexKey`	`[]byte`	`nil`	32-byte encryption key (SDK-supplied KEK). Set either `IndexKey` or `KmsName`, never both.
`KmsName`	`*string`	`nil`	Name of a `kms.registry` entry in the service YAML. The service generates the KEK and wraps it with that provider; the SDK never sees the plaintext key.
`Dimension`	`*int32`	`nil`	Vector dimensionality. When `nil`, the service detects it from the first upsert.
`Metric`	`*string`	`nil` (service default: `"euclidean"`)	`"euclidean"`, `"squared_euclidean"`, or `"cosine"`.
`EmbeddingModel`	`*string`	`nil`	Name of one of the service’s built-in embedding models (listed by `GET /v1/embedding-models`), used for automatic embedding generation. When set, `Dimension` is taken from the model.
`StoragePrecision`	`*string`	`nil` (service default: `"float32"`)	Storage format for the index’s stored vector embeddings. See Storage precision.
`MetadataSchema`	`map[string]MetadataFieldPolicy`	`nil`	Per-field metadata indexing policy. Fixed at creation and cannot be changed later. See Metadata schema.
`TextFields`	`[]string`	`nil`	Metadata fields to index for BM25 full-text search. Shorthand for `FullText: true` in `MetadataSchema`. Enables `Text` on `Query` and `QueryMetadata`.
`Bm25K1`	`*float64`	`nil` (service default: `1.2`)	BM25 term-frequency saturation, `>= 0`. Requires at least one full-text field.
`Bm25B`	`*float64`	`nil` (service default: `0.75`)	BM25 length-normalization strength, in `[0, 1]`. Requires at least one full-text field.

Key management

Set exactly one of IndexKey or KmsName. The service rejects both together with a *ValidationError (HTTP 400), whatever the KMS provider.

SDK-supplied KEK: set IndexKey. Pass the same key again on every LoadIndex.
KMS-backed: set KmsName. The service generates the key, wraps it with the named provider, and stores the wrapped key. Later calls pass a nil key.

Storage precision

StoragePrecision sets the format used to store the index’s vector embeddings. Every value works with every metric.

Constant	Value	Bits per dimension
`StoragePrecisionFloat32`	`"float32"`	32
`StoragePrecisionFloat16`	`"float16"`	16
`StoragePrecisionTQ12`	`"tq12"`	12
`StoragePrecisionTQ8`	`"tq8"`	8
`StoragePrecisionTQ6`	`"tq6"`	6
`StoragePrecisionTQ4`	`"tq4"`	4

The tq* values are TurboQuant tiers; lower tiers use less storage in exchange for a small recall and latency cost.

Metadata schema

MetadataSchema sets how each metadata field is indexed. Metadata itself stays schemaless: this controls indexing, not what may be stored. Fields left out get the default policy (indexed and filterable). Nested fields use dot-paths such as "loc.city". MetadataFieldPolicy has three *bool fields; nil means the default:

Field	Default	Effect
`Filterable`	`true`	Indexes the field, so filters on it resolve from the index. `false` still stores the field but doesn’t index it; see Filters the index cannot answer.
`Pattern`	`false`	Also builds a regex dictionary, so `$regex` and `$contains` resolve from the index. Requires `Filterable` `true`.
`FullText`	`false`	Sends the field’s text through the BM25 analyzer instead of exact-match indexing. Implies `Filterable` `false`, and can’t be combined with `Pattern` `true`.

Filters the index cannot answer

The index can answer a filter when the field is filterable and, for $regex or $contains, the field also has pattern set. Any other filter is handled differently depending on the call:

Call	Filter the index can’t answer
`Query` without `text`	Still works, but slower: the filter is applied to the decrypted metadata after the search.
`Query` with `text`	Rejected with a `*ValidationError`.
`QueryMetadata`	Rejected with a `*ValidationError`.

With the schema from the per-field example below:

map[string]interface{}{"title": map[string]interface{}{"$regex": "^Intro"}}  // answerable: title is filterable and has Pattern
map[string]interface{}{"internal_blob": "abc"}                               // not answerable: internal_blob isn't filterable
map[string]interface{}{"author": map[string]interface{}{"$regex": "^Ann"}}   // not answerable: author is filterable by default, but has no Pattern

The stored policy is returned by the MetadataSchema getter.

Returns

The *EncryptedIndex bound to the new index. For KMS-backed indexes it holds no plaintext key.

Errors

*ValidationError

Wraps ErrNilParams, ErrMissingKeyOrKMS, or ErrInvalidKeyLength. params is nil, neither IndexKey nor KmsName is set, or IndexKey isn’t 32 bytes. Returned before any request is sent.
The service rejects the request (HTTP 400/422): the index name already exists, both IndexKey and KmsName are set, the KmsName slot is unknown, StoragePrecision isn’t one of the six values, or MetadataSchema combines settings that aren’t allowed.

*AuthenticationError

The client’s apiKey is missing or invalid (HTTP 401), or is a per-user key: creating an index requires the root key (HTTP 403).

*ServiceError

The service returns 5xx, including KMS wrap failures (502).

*TransportError

The service can’t be reached.

See Errors.

Example Usage

SDK-supplied key

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/cyborginc/cyborgdb-go"
)

func main() {
    client, err := cyborgdb.NewClient("http://localhost:8000", "your-api-key")
    if err != nil {
        log.Fatal(err)
    }

    key, err := cyborgdb.GenerateKey()
    if err != nil {
        log.Fatal(err)
    }

    // The dimension is detected from the first upsert
    index, err := client.CreateIndex(context.Background(), &cyborgdb.CreateIndexParams{
        IndexName: "documents",
        IndexKey:  key,
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(index.GetIndexName())
    // Output: documents
}

Store the key securely: it is needed to load the index again. See Generate Key.

Explicit dimension and metric

dimension := int32(384)
metric := "cosine"

index, err := client.CreateIndex(ctx, &cyborgdb.CreateIndexParams{
    IndexName: "documents-384",
    IndexKey:  key,
    Dimension: &dimension,
    Metric:    &metric,
})
if err != nil {
    log.Fatal(err)
}
_ = index

KMS-backed index

// The service generates the key and wraps it with the "prod-aws" registry slot.
kmsName := "prod-aws"
dimension := int32(384)

index, err := client.CreateIndex(ctx, &cyborgdb.CreateIndexParams{
    IndexName: "kms-documents",
    KmsName:   &kmsName,
    Dimension: &dimension,
})
if err != nil {
    log.Fatal(err)
}
_ = index
// Later calls, including LoadIndex, pass a nil key.

Automatic embeddings

// The dimension is taken from the embedding model
model := "all-MiniLM-L6-v2"

index, err := client.CreateIndex(ctx, &cyborgdb.CreateIndexParams{
    IndexName:      "semantic-docs",
    IndexKey:       key,
    EmbeddingModel: &model,
})
if err != nil {
    log.Fatal(err)
}
_ = index

See Auto-Generate Embeddings.

Reduced storage with TurboQuant

dimension := int32(768)
precision := cyborgdb.StoragePrecisionTQ8

index, err := client.CreateIndex(ctx, &cyborgdb.CreateIndexParams{
    IndexName:        "compact-docs",
    IndexKey:         key,
    Dimension:        &dimension,
    StoragePrecision: &precision,
})
if err != nil {
    log.Fatal(err)
}
_ = index

Per-field metadata policy

index, err := client.CreateIndex(ctx, &cyborgdb.CreateIndexParams{
    IndexName: "docs-index",
    IndexKey:  key,
    Dimension: cyborgdb.Int32(4),
    MetadataSchema: map[string]cyborgdb.MetadataFieldPolicy{
        "title":         {Filterable: cyborgdb.Bool(true), Pattern: cyborgdb.Bool(true)},
        "internal_blob": {Filterable: cyborgdb.Bool(false)},
    },
})
if err != nil {
    log.Fatal(err)
}

err = index.Upsert(ctx, cyborgdb.VectorItems{
    {Id: "doc1", Vector: []float32{0.1, 0.2, 0.3, 0.4}, Metadata: map[string]interface{}{"title": "intro to encryption"}},
    {Id: "doc2", Vector: []float32{0.5, 0.6, 0.7, 0.8}, Metadata: map[string]interface{}{"title": "advanced setup"}},
})
if err != nil {
    log.Fatal(err)
}

// $regex resolves from the index because "title" has Pattern set
res, err := index.QueryMetadata(ctx, cyborgdb.QueryMetadataParams{
    Filters: map[string]interface{}{"title": map[string]interface{}{"$regex": "^intro"}},
})
if err != nil {
    log.Fatal(err)
}
fmt.Println(res.Ids)
// Output: [doc1]

Full-text search fields

dimension := int32(4)

index, err := client.CreateIndex(ctx, &cyborgdb.CreateIndexParams{
    IndexName:  "articles",
    IndexKey:   key,
    Dimension:  &dimension,
    TextFields: []string{"body"},
})
if err != nil {
    log.Fatal(err)
}

bm25, err := index.BM25(ctx)
if err != nil {
    log.Fatal(err)
}
fmt.Println(bm25.K1, bm25.B, bm25.GetAnalyzerVersion())
// Output: 1.2 0.75 en/1;sw=2de831ef7be676b6;snowball-english-utf8-3.0.1

Introduction

Client

Encrypted Index

Types & Helpers

Parameters

CreateIndexParams

Key management

Storage precision

Metadata schema

Filters the index cannot answer

Returns

Errors

Example Usage

SDK-supplied key

Explicit dimension and metric

KMS-backed index

Automatic embeddings

Reduced storage with TurboQuant

Per-field metadata policy

Full-text search fields

​Parameters

​CreateIndexParams

​Key management

​Storage precision

​Metadata schema

​Filters the index cannot answer

​Returns

​Errors

​Example Usage

​SDK-supplied key

​Explicit dimension and metric

​KMS-backed index

​Automatic embeddings

​Reduced storage with TurboQuant

​Per-field metadata policy

​Full-text search fields

Parameters

CreateIndexParams

Key management

Storage precision

Metadata schema

Filters the index cannot answer

Returns

Errors

Example Usage

SDK-supplied key

Explicit dimension and metric

KMS-backed index

Automatic embeddings

Reduced storage with TurboQuant

Per-field metadata policy

Full-text search fields