Data Catalog and Secrets | Kamiwaza Docs

Kamiwaza provides a centralized catalog for datasets, containers, and secrets. The catalog stores metadata, supports DataHub URNs, and is protected by access controls so only authorized users can view or change assets.

In current deployments, the catalog is the standard place to reference shared data assets and secret metadata across models, connectors, applications, and tools.

Core entities

Datasets

Datasets are the primary catalog entity. They are identified by DataHub URNs and can include schema metadata.

Containers

Containers are logical groupings of datasets (for example, a project or domain). They also use DataHub URNs and support dataset membership APIs. They also map well to things like S3 buckets, file folders, etc.

Secrets

Secrets are stored as catalog entities with encrypted values. Secret metadata is accessible via the API, but secret values are never returned through HTTP endpoints.

Secrets are encrypted with a dual DEK/KEK scheme, with support for secret rotation.

URN handling (DataHub)

Some DataHub URNs include forward slashes, which can conflict with path routing. Kamiwaza supports three approaches:

  1. V2 regex path endpoints (recommended)
  2. Query-parameter endpoints (/by-urn)
  3. Legacy path endpoints (simple URNs only)

Dataset endpoints

GET    /api/catalog/datasets/v2/{dataset_urn}

GET    /api/catalog/datasets/v2/{dataset_urn}/schema

PATCH  /api/catalog/datasets/v2/{dataset_urn}

DELETE /api/catalog/datasets/v2/{dataset_urn}

PUT    /api/catalog/datasets/v2/{dataset_urn}/schema

GET    /api/catalog/datasets/by-urn?urn={full_urn}

GET    /api/catalog/datasets/by-urn/schema?urn={full_urn}

PATCH  /api/catalog/datasets/by-urn?urn={full_urn}

DELETE /api/catalog/datasets/by-urn?urn={full_urn}

PUT    /api/catalog/datasets/by-urn/schema?urn={full_urn}

GET    /api/catalog/datasets/{encoded_urn}

PATCH  /api/catalog/datasets/{encoded_urn}

DELETE /api/catalog/datasets/{encoded_urn}

Container endpoints

GET    /api/catalog/containers/v2/{container_urn}

PATCH  /api/catalog/containers/v2/{container_urn}

DELETE /api/catalog/containers/v2/{container_urn}

POST   /api/catalog/containers/v2/{container_urn}/datasets

DELETE /api/catalog/containers/v2/{container_urn}/datasets/{dataset_urn}

GET    /api/catalog/containers/by-urn?urn={full_urn}

PATCH  /api/catalog/containers/by-urn?urn={full_urn}

DELETE /api/catalog/containers/by-urn?urn={full_urn}

Secret endpoints

POST   /api/catalog/secrets

GET    /api/catalog/secrets

GET    /api/catalog/secrets/v2/{secret_urn}

DELETE /api/catalog/secrets/v2/{secret_urn}

GET    /api/catalog/secrets/by-urn?urn={full_urn}

DELETE /api/catalog/secrets/by-urn?urn={full_urn}

Note: Secret values are never exposed in API responses. Use secrets in connectors and deployments by reference.

Access control

Best practices