Metadata Filtering

CyborgDB supports metadata tagging and filtering, enabling fine-grained search filters in addition to encrypted vector search. Metadata, like vectors, are end-to-end encrypted and never exposed in plaintext outside of the client.

Supported Metadata

The following metadata types are supported:

Boolean
String
List of strings
Numbers (will be stored as fp64)

Adding Items with Metadata

To add items with metadata tags, you can pass a dictionary of key-value pairs to the metadata field during upsert:

# Example data
data = [
    {"id": "item_1", "vector": [0.1, 0.1, 0.1, 0.1], "metadata": {"category": "dog"}},
    {"id": "item_2", "vector": [0.2, 0.2, 0.2, 0.2], "metadata": {"category": "cat"}}
]

# Upsert data with metadata
index.upsert(data)

// Example data
const data = [
    {id: "item_1", vector: [0.1, 0.1, 0.1, 0.1], metadata: {category: "dog"}},
    {id: "item_2", vector: [0.2, 0.2, 0.2, 0.2], metadata: {category: "cat"}}
];

// Upsert data with metadata
await index.upsert({ items: data });

import { VectorItem } from 'cyborgdb';

// Example data
const data: VectorItem[] = [
    {id: "item_1", vector: [0.1, 0.1, 0.1, 0.1], metadata: {category: "dog"}},
    {id: "item_2", vector: [0.2, 0.2, 0.2, 0.2], metadata: {category: "cat"}}
];

// Upsert data with metadata
await index.upsert({ items: data });

package main

import (
    "context"
    "github.com/cyborginc/cyborgdb-go"
)

// Example data
items := cyborgdb.VectorItems{
    {
        Id:       "item_1",
        Vector:   []float32{0.1, 0.1, 0.1, 0.1},
        Metadata: map[string]interface{}{"category": "dog"},
    },
    {
        Id:       "item_2", 
        Vector:   []float32{0.2, 0.2, 0.2, 0.2},
        Metadata: map[string]interface{}{"category": "cat"},
    },
}

// Upsert data with metadata
err := index.Upsert(context.Background(), items)

curl -X POST "http://localhost:8000/v1/vectors/upsert" \
     -H "X-API-Key: your-api-key" \
     -H "Content-Type: application/json" \
     -d '{
       "index_name": "my_index",
       "index_key": "your_64_character_hex_key_here",
       "items": [
         {
           "id": "item_1",
           "vector": [0.1, 0.1, 0.1, 0.1],
           "metadata": {"category": "dog"}
         },
         {
           "id": "item_2",
           "vector": [0.2, 0.2, 0.2, 0.2],
           "metadata": {"category": "cat"}
         }
       ]
     }'

This metadata will be encrypted and stored in the index.

Metadata field names and string values are case-sensitive. Ensure that the metadata field names and values are consistent when filtering queries.

Filtering Queries with Metadata

CyborgDB supports a subset of MongoDB’s Query and Projection Operators. For more details, see Metadata Query Operators To filter a query with metadata, you can pass a dictionary of filters to the filters field during query:

# Example query
query_vector = [0.5, 0.9, 0.2, 0.7]
top_k = 10

# Example filters
filters = {
    "category": {"$in": ["dog", "cat"]} # Will match either 'dog' or 'cat'
}

# Perform query
results = index.query(
    query_vectors=query_vector, 
    top_k=top_k, 
    filters=filters
)

// Example query
const queryVectors = [0.5, 0.9, 0.2, 0.7];
const topK = 10;

// Example filters
const filters = {
    category: {$in: ["dog", "cat"]} // Will match either 'dog' or 'cat'
};

// Perform query
const results = await index.query({
    queryVectors,
    topK,
    filters
});

import { QueryResponse, FilterExpression } from 'cyborgdb';

// Example query
const queryVectors: number[] = [0.5, 0.9, 0.2, 0.7];
const topK: number = 10;

// Example filters
const filters: FilterExpression = {
    category: {$in: ["dog", "cat"]} // Will match either 'dog' or 'cat'
};

// Perform query
const params = {
    queryVectors,
    topK,
    filters
};
const response: QueryResponse = await index.query(params);

// Example query
queryVector := []float32{0.5, 0.9, 0.2, 0.7}
topK := int32(10)

// Example filters
filters := map[string]interface{}{
    "category": map[string]interface{}{"$in": []string{"dog", "cat"}}, // Will match either 'dog' or 'cat'
}

// Perform query
results, err := index.Query(context.Background(), cyborgdb.QueryParams{
    QueryVector: queryVector,
    TopK:        topK,
    Filters:     filters,
})

curl -X POST "http://localhost:8000/v1/vectors/query" \
     -H "X-API-Key: your-api-key" \
     -H "Content-Type: application/json" \
     -d '{
       "index_name": "my_index",
       "index_key": "your_64_character_hex_key_here",
       "query_vectors": [0.5, 0.9, 0.2, 0.7],
       "top_k": 10,
       "filters": {
         "category": {"$in": ["dog", "cat"]}
       }
     }'

Metadata Indexing

By default, every metadata field is indexed using encrypted indexing, so filters on indexed fields are resolved from the index and are fast. You can opt individual fields out of indexing with metadata_schema at index creation (see Metadata schema). Filters on non-indexed fields still work, but they are evaluated after decryption and are slower, as are $regex / $contains filters on fields not indexed for pattern matching.

Metadata Query Operators

CyborgDB supports a subset of MongoDB’s Query and Projection Operators. Specifically, the following operators are supported:

Filter	Types	Description
`$and`	`array`	Logical AND - matches vectors that satisfy all conditions in the array
`$or`	`array`	Logical OR - matches vectors that satisfy at least one condition in the array
`$not`	`object`	Logical NOT - matches vectors that do not satisfy the condition
`$exists`	`boolean`	Matches vectors that have (or do not have) the specified field
`$eq`	`Boolean`, `Number`, `String`	Matches vectors with the metadata that is equal to the filter value
`$ne`	`Boolean`, `Number`, `String`	Matches vectors with the metadata that is not equal to the filter value
`$in`	`Number`, `String`	Matches vectors with metadata that is in the filter array
`$nin`	`Number`, `String`	Matches vectors with metadata that is not in the filter array
`$gt`	`Number`	Matches vectors with metadata that is greater than the filter value
`$gte`	`Number`	Matches vectors with metadata that is greater than or equal to the filter value
`$lt`	`Number`	Matches vectors with metadata that is less than the filter value
`$lte`	`Number`	Matches vectors with metadata that is less than or equal to the filter value

CyborgDB Service

Encrypted Indexes

Data Operations

Advanced

Supported Metadata

Adding Items with Metadata

Filtering Queries with Metadata

Metadata Indexing

Metadata Query Operators

​Supported Metadata

​Adding Items with Metadata

​Filtering Queries with Metadata

​Metadata Indexing

​Metadata Query Operators

Supported Metadata

Adding Items with Metadata

Filtering Queries with Metadata

Metadata Indexing

Metadata Query Operators