Managing Encryption Keys

Encryption keys are one of the most critical elements that enable the security guarantees of CyborgDB. By default, one key is used for the entire index, and it is used to generate and secure other encryption keys, including:

Encrypted index tokens
Index centroid salted hashes
Index payload (embedding) encryption
Item content encryptions
Encrypted query tokens

This main key, known in the code as the index_key, is a 256-bit (32-byte) symmetric Key-Encryption-Key (KEK)—the same you would find when using AES-256 encryption. CyborgDB wraps a per-index data key (DEK) under this KEK internally and encrypts all index data with that DEK using AES-256-GCM; callers only ever handle the KEK. CyborgDB’s cryptography is based entirely on well-established cryptographic standards, including AES-256-GCM, HMAC, and SHA-3 (Keccak). Without the correct index_key, it is impossible to use a CyborgDB Encrypted Index. You cannot upsert vectors, query the index, or even delete it. Hence, it is critical to manage these encryption keys safely.

For multi-tenant scenarios, you can mint per-user keys with scoped read/write permissions instead of sharing the root KEK — see Access Control (RBAC). To persist an external-KMS wrapping envelope for the KEK, see External Key Management (KMS).

Generating Key Locally (for Development)

Do not store keys locally for production purposes. For production environments, always use secure options such as Hardware Security Modules (HSMs) or a Key Management Service (KMS). See Using Key Management Services (for Production).

For local development and evaluation, you can generate a 256-bit encryption key locally and use it in your calls to CyborgDB. Here’s how you can do this using OpenSSL and integrate it with CyborgDB in Python. The index_key must be exactly 32 raw bytes. The simplest way to generate one is with the language’s cryptographic RNG.

Generate a 32-byte key

import secrets

index_key = secrets.token_bytes(32)  # 32-byte index KEK

#include <array>
#include <openssl/rand.h>

std::array<uint8_t, 32> index_key;
RAND_bytes(index_key.data(), index_key.size());  // 32-byte index KEK

Alternatively, generate a key on the command line and persist the raw bytes to a file:

openssl rand 32 > index_key.bin

This saves 32 raw bytes to index_key.bin. Ensure that this file is kept secure and not included in your source control.

Use the key with CyborgDB

Pass the key directly to create_index (or load_index):

import cyborgdb_core as cyborgdb

api_key = "your_api_key_here"  # Replace with your CyborgDB API key

# Backing store: in-memory (use .disk("/path") or .s3("bucket") for persistence)
client = cyborgdb.Client(api_key, cyborgdb.StorageConfig.memory())

# Load the raw 32-byte key (or use the one generated above)
with open("index_key.bin", "rb") as key_file:
    index_key = key_file.read()

# Use the encryption key
index = client.create_index("test_index", index_key)

#include "cyborgdb_core/client.hpp"
#include "cyborgdb_core/encrypted_index.hpp"

std::string api_key = "your_api_key_here";  // Replace with your CyborgDB API key

// In-memory backing store (use StorageConfig::Disk(path) or ::S3(bucket) to persist)
cyborg::Client client(api_key, cyborg::StorageConfig::Memory(), 0, cyborg::kNone);

// index_key holds the 32 raw bytes generated above
auto index = client.CreateIndex("test_index", index_key);

Again, this should only be used for development and evaluation purposes; do not store encryption keys locally for production use.

If you don’t have an API key yet, see Get an API Key.

Using Key Management Services (for Production)

For production environments, a Key Management Service (KMS) or Hardware Security Module (HSM) is strongly recommended to securely store and manage encryption keys. This ensures the highest level of security by isolating the keys from application logic and providing robust access controls.

Overview of KMS

A KMS is a managed service that provides centralized control over encryption keys, including:

Key Generation: Automatically generating keys with required levels of entropy.
Secure Storage: Storing keys in a tamper-proof environment.
Access Control: Enforcing strict role-based access control (RBAC) policies.
Auditing: Logging key usage for compliance and monitoring.

Supported KMS Providers

CyborgDB can integrate with popular KMS providers, such as:

AWS Key Management Service (AWS KMS)
Google Cloud Key Management (Google Cloud KMS)
Azure Key Vault
HashiCorp Vault

Step-by-Step Example: AWS KMS

Generate a Data Key Using AWS KMS

Option A: Command LineUse the AWS CLI to generate a data key:

aws kms generate-data-key \
    --key-id <Your-Key-Id> \
    --key-spec AES_256 \
    --output json

Replace <Your-Key-Id> with your KMS Key ID or ARN.
The command outputs a JSON object containing both the plaintext key and the encrypted key.

Example Output:

{
"CiphertextBlob": "<Base64-encoded encrypted key>",
"Plaintext": "<Base64-encoded plaintext key>",
"KeyId": "arn:aws:kms:region:account-id:key/Your-Key-Id"
}

Note: The Plaintext and CiphertextBlob are Base64-encoded binary data.Option B: Python

import boto3

# Initialize AWS KMS client
kms_client = boto3.client('kms')

# Generate a data key
response = kms_client.generate_data_key(
    KeyId='Your-Key-Id',  # Replace with your KMS Key ID or ARN
    KeySpec='AES_256'
)

# Extract the plaintext and encrypted data key
plaintext_key = response['Plaintext']       # This is the raw binary key
encrypted_key = response['CiphertextBlob']  # This is the encrypted key

Store the Encrypted Key

Encrypted Key: Since it’s encrypted, you can safely store it in your application’s configuration file, environment variable, or secure parameter store.

Example of storing the encrypted key in Base64-encoded format:

import base64

# Base64-encode the encrypted key for storage
encrypted_key_b64 = base64.b64encode(encrypted_key).decode('utf-8')

# Store 'encrypted_key_b64' in your application's configuration

Use the Encrypted Key in Your Application at Runtime

In your application code, decrypt the encrypted key at runtime using AWS KMS:

import boto3
import base64
import cyborgdb_core as cyborgdb

api_key = "your_api_key_here"  # Replace with your CyborgDB API key

# Backing store: in-memory (use .disk("/path") or .s3("bucket") for persistence)
client = cyborgdb.Client(api_key, cyborgdb.StorageConfig.memory())

# Initialize AWS KMS client
kms_client = boto3.client('kms')

# Load the encrypted key from your configuration
encrypted_key_b64 = '...'  # Replace with your stored encrypted key (Base64-encoded)

# Decode the Base64-encoded encrypted key
encrypted_key = base64.b64decode(encrypted_key_b64)

# Decrypt the key at runtime — yields the raw 32-byte KEK
response = kms_client.decrypt(
    CiphertextBlob=encrypted_key
)
index_key = response['Plaintext']  # This is your 256-bit (32-byte) KEK

# Use the key with CyborgDB
index = client.create_index("test_index", index_key)

Explanation:

Encrypted Key Storage: The encrypted key (encrypted_key_b64) is safe to store in configuration files since it’s encrypted.
Runtime Decryption: The kms_client.decrypt method securely retrieves the plaintext key at runtime.

Ensure IAM Permissions

Your application needs permission to use the kms:Decrypt action. Attach a policy to your IAM role or user:

{
"Version": "2012-10-17",
"Statement": [
    {
    "Effect": "Allow",
    "Action": "kms:Decrypt",
    "Resource": "arn:aws:kms:your-region:your-account-id:key/Your-Key-Id"
    }
]
}

Benefits of This Approach

Security: The plaintext key is never stored persistently. It’s only available in memory during runtime.
Simplicity: You can generate and manage the key using AWS KMS and standard AWS tools.
No Need for Secure Storage: Since the encrypted key is safe to store in your application’s configuration, you don’t need additional secure storage solutions.

By generating the data key using AWS KMS and storing only the encrypted key, you can securely manage your encryption key without needing to store it securely elsewhere. Your application decrypts the encrypted key at runtime using AWS KMS, ensuring that the plaintext key is only available in memory during execution.

CyborgDB can also persist a per-index KMS envelope describing how the KEK is wrapped, so a service layer can unwrap it at request time. See External Key Management (KMS) for details on KMSBlob and the create_index_kms / push_index_kms / get_index_kms / delete_index_kms functions.

Intro

Encrypted Indexes

Data Operations

Advanced

Managing Encryption Keys

Generating Key Locally (for Development)

Using Key Management Services (for Production)

Overview of KMS

Supported KMS Providers

Step-by-Step Example: AWS KMS

Benefits of This Approach

​Generating Key Locally (for Development)

​Using Key Management Services (for Production)

​Overview of KMS

​Supported KMS Providers

​Step-by-Step Example: AWS KMS

​Benefits of This Approach

Generating Key Locally (for Development)

Using Key Management Services (for Production)

Overview of KMS

Supported KMS Providers

Step-by-Step Example: AWS KMS

Benefits of This Approach