Skip to main content
CyborgDB Service v0.18 supports per-index Key Management Service (KMS) integration: each index can be wrapped by its own operator-managed key, optionally living in a customer’s own AWS account (Bring Your Own KMS). The SDK never holds long-term encryption keys for KMS-backed indexes.
This page is service-only. The embedded libraries’ KMS story is covered separately under Managing Encryption Keys.
When KMS-backed, omit index_key everywhere. Migrating an index from the SDK-supplied path to KMS-backed means removing index_key from every call — create_index, load_index, describe_index, train, upsert, query, get, delete, list_ids, query_metadata, delete_index, and the user-management calls. The service resolves the KEK server-side from the index’s stored key envelope; supplying index_key for a KMS-backed index is rejected with HTTP 400. If your code still passes a key, the rejection is loud — but worth a single search-and-clean pass before you ramp.

AWS access — what the service expects

The service uses the standard AWS credential provider chain to reach aws-kms and aws (Secrets Manager) slots. The exact resolution depends on where you run it:
The CYBORGDB_S3_* storage credentials are deliberately separate from AWS_* KMS credentials. KMS uses the AWS default chain (or AssumeRole); S3 storage uses its own explicit keys when both CYBORGDB_S3_ACCESS_KEY and CYBORGDB_S3_SECRET_KEY are set. Set both for a non-AWS storage backend (MinIO, R2, etc.); if either is unset, storage falls back to the same AWS default chain that KMS uses.

Required IAM permissions (per provider)

provider: aws-kms — on the wrap key:
For same-account setups, these permissions can live on the KMS key’s key policy (preferred) or on the runtime principal’s IAM policy. provider: aws (Secrets Manager) — on the secret:
Secrets Manager appends 6 random characters to every secret ARN, so either use the secret’s full ARN as-is, or the ARN up to the secret name followed by -* as shown.

Model

CyborgDB Service uses a two-key hierarchy per index:
  • KEK (Key-Encryption Key, 32 bytes) — per index. Resolved from the named KMS registry slot at index creation, then re-fetched (or unwrapped) on every load. Held in a short-lived in-process cache; the TTL is set by INDEX_KEK_CACHE_TTL_SECONDS (default 60 s).
  • DEK (Data-Encryption Key, 32 bytes) — per index. Generated internally at index-creation time, wrapped under the KEK with AES-GCM, and persisted alongside the index. Never leaves the service and is never returned to clients.
There are exactly two ways to provision a CyborgDB index: Supplying both kms_name and index_key is rejected with a 400. Supplying neither is also a 400.

YAML Registry

Per-index KMS slots live in the service YAML file (not environment variables) under kms.registry. Each slot is a named entry that create_index(..., kms_name=<slot>) can reference.
cyborgdb.yaml
String values support env-var substitution: ${VAR} (required, fails on startup if unset) and ${VAR:-default} (uses the default when unset). Use this to keep BYOK role ARNs and external IDs out of the checked-in file.
role_session_name defaults to cyborgdb-<key_id> (with any alias/ prefix removed). STS rejects session names containing / or :, so set role_session_name explicitly on any BYOK slot whose key_id is a path-style name (such as customers/acme/kek) or an ARN.

Provider types

The provider field selects how the KEK is wrapped: Both providers additionally accept role_arn + external_id for cross-account access (BYOK) — the service calls sts:AssumeRole before reaching the key. There is no registry slot for the SDK-supplied path. Omit kms_name entirely from create_index and supply index_key directly; the persisted envelope records provider: none.

Caching and revocation

The service caches plaintext KEKs in memory for INDEX_KEK_CACHE_TTL_SECONDS (default 60 s). Only KMS-derived KEKs are cached — the SDK-supplied path always passes through. Shorter TTLs propagate KMS revocations (key deletion, IAM policy detach, trust-policy edit) faster but cost more KMS calls. To force-revoke an index globally, revoke the wrap key in the KMS provider; cached KEKs expire within INDEX_KEK_CACHE_TTL_SECONDS.

Creating a KMS-backed index

The SDK call is unchanged except for swapping index_key= for kms_name=:
On every subsequent request for this index — load_index, upsert, query, delete — the SDK omits index_key. The service looks up the index’s persisted envelope, resolves the KEK via the named KMS slot (cache hit, or fresh wrap/unwrap on miss), and uses it to open the index.

Bring Your Own KMS (cross-account)

In BYOK, the wrap key lives in the customer’s AWS account. CyborgDB Service holds no long-term credentials to that account — access flows through sts:AssumeRole with an ExternalId on every wrap or unwrap call. Setup is a two-party handshake:
1

Service operator: generate an ExternalId

Treat as a credential — it’s the cryptographic gate preventing one customer’s role ARN from being abused by another.
2

Service operator: share three values with the customer

  • The service’s AWS principal ARN (the service’s AWS identity). Get it with:
  • The ExternalId you generated.
  • The customer-facing setup steps (Step 3 below).
3

Customer: create the wrap key and IAM role

1. Create the wrap key in Secrets Manager (32 random bytes):
Keep a backup. If this secret is deleted, every index wrapped under it becomes permanently unreadable.
2. Create an IAM role with this trust policy (substituting the operator-supplied values):
3. Attach an inline permission policy for the secret:
Secrets Manager appends 6 random chars to every secret ARN, so either use the secret’s full ARN as-is, or the ARN up to the secret name followed by -* as shown.Send three values back to the operator: role ARN, secret name, region. Never share AWS credentials.
4

Service operator: add the slot to YAML

Under kms.registry::
Then restart the service. On boot, look for:
5

(Optional) Verify the AssumeRole

From the same shell environment as the service:
Prints ASIA... → trust policy and ExternalId match. AccessDenied → the customer’s trust policy is wrong; share the error.

Revocation

  • Pause access — customer detaches the inline permission policy from the role. The service starts failing on the next cache miss (i.e. within INDEX_KEK_CACHE_TTL_SECONDS).
  • Permanent revoke — customer deletes the role or the secret. Note: deleting the secret renders every index wrapped under it permanently unreadable.

Configuration changes after creation

Changes to the YAML registry take effect only after a service restart. Each index remembers the provider, key_id, and region it was created with, and on restart the service handles changes as follows:
  • role_arn / external_id / role_session_name changes — picked up automatically. Nothing else to do.
  • provider / key_id / region changes — treated as a key rotation. The service re-wraps each affected index under the new key on startup. The old key must still be reachable with the slot’s new role_arn / external_id, so change the account and the key in separate steps. If a rotation fails, the service still starts, but requests for that index fail until the issue is fixed; check the startup logs.
  • Slot removed — don’t remove a slot while indexes still use it. Requests for those indexes fail with HTTP 401 until the slot is restored.

Troubleshooting

Trust policy mismatch. Either the Principal doesn’t match the service’s AWS identity, or sts:ExternalId doesn’t match the operator-provided UUID. Re-run the manual verify in Step 5 above and share the exact error with the customer.
The permission policy on the customer’s role doesn’t cover the secret, or the Resource ARN is missing the required -* suffix (Secrets Manager appends 6 random chars).
The secret was deleted, or the region in the YAML slot doesn’t match the region where the secret lives. Seen as HTTP 502 on index creation (...; cannot wrap) and HTTP 401 on later requests (...; cannot unwrap).
The Secrets Manager value is the wrong size. Replace with exactly 32 random bytes.
Supplying both fields is rejected with HTTP 400 regardless of provider type. Pick one path per index. Supplying neither is also a 400: exactly one of kms_name or index_key must be supplied.
A request for a KMS-backed index included index_key (HTTP 400). Remove it from the call.
kms_name doesn’t match any slot under kms.registry (HTTP 400). The error lists the known slots. Check the startup log line KMS registry loaded (...).
Plaintext KEKs are cached for INDEX_KEK_CACHE_TTL_SECONDS (default 60 s). For tighter revocation windows, drop the TTL — at the cost of more KMS calls. There is no live invalidation API in v0.18.

See also

  • Environment Variables — full configuration reference, including INDEX_KEK_CACHE_TTL_SECONDS.
  • Managing Encryption Keys — the SDK-supplied (provider: none) path and the legacy “client decrypts via KMS” pattern.
  • Create Index — the kms_name parameter and the SDK-supplied alternative.