Federated Retrieval | Kamiwaza Docs

This is documentation for Kamiwaza 1.0.1, which is no longer actively maintained. For the current GA release, see 1.0.1.

Version: 1.0.1

Federated Retrieval

Federated retrieval lets you discover and query data on remote clusters without the data leaving its origin. The mesh proxy routes your requests transparently — the remote cluster's catalog and retrieval services process them as if they were local.

Prerequisites

• A paired federation between the clusters
• Remote CA cert stored in the federation record
• The mesh user must have per-dataset ReBAC grants on the target cluster (admin users with extauthz trust are exempt)

Discover Remote Datasets

# List datasets on the remote cluster

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/catalog/datasets/" \
  -H "Authorization: Bearer $TOKEN"

Response: array of dataset metadata (URNs, names, schemas) from the remote cluster's catalog.

Retrieve Remote Data

# Create a retrieval job on the remote cluster

curl -sk -X POST "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_urn": "urn:li:dataset:(urn:li:dataPlatform:postgres,nps_verbatims,PROD)",
    "columns": ["customer_id", "verbatim", "score"],
    "filters": {"quarter": "Q1-2026"},
    "transport": "inline"
  }'

The retrieval service on the remote cluster accesses its local data adapters (S3, Postgres, Hive, etc.) and returns the results through the mesh proxy. Only the query results cross the network — raw data stays on the remote cluster.

Streaming Retrieval (SSE)

# Submit job, then stream results

JOB_ID=$(curl -sk -X POST "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataset_urn": "...", "transport": "sse"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["job_id"])')

# Stream results via SSE

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs/$JOB_ID/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

SSE events stream through the mesh proxy transparently.

List Remote Models

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/serving/deployments" \
  -H "Authorization: Bearer $TOKEN"

Returns deployment metadata: model names, engine types, status, and access paths.

Run Remote Inference

# Chat completion on a remote model

curl -sk -X POST \
  "https://kamiwaza.test/api/mesh/<remote-cluster>/runtime/models/<deployment-id>/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-4B",
    "messages": [{"role": "user", "content": "Summarize the NPS themes"}]
  }'

The inference request routes through the mesh proxy to the remote cluster's Ray Serve deployment. The response streams back.

Access Control

Admin Users

On Istio clusters with the extauthz trust model, admin users bypass per-resource ReBAC checks on the target cluster (standard admin bypass). No additional seeding is needed.

Non-Admin Users

Non-admin users require explicit per-dataset ReBAC grants on the target cluster. The admin bypass is suppressed for mesh-originated requests — federated callers must have explicit viewer access to each dataset they query or browse. The catalog listing endpoint filters results so federated users only see datasets they have grants for. Grant access at federation-pairing time through the brokered-user allowlist's initial_tuples field — not through /api/auth/tuples. Why not /api/auth/tuples? A federated caller has no local user account on the target cluster until their first mesh request, when the brokering service auto-provisions a local Keycloak user with a freshly-minted UUID. The per-dataset check authorizes against that local UUID, which isn't known ahead of time — so a tuple written against the source cluster's user UUID via /api/auth/tuples never matches, and the call returns 204 while access stays denied (404 Dataset not found). The allowlist's initial_tuples solves this with a {{user_id}} placeholder that renders to the local UUID at provision time.

# On the TARGET cluster, add the source user to the federation allowlist

# WITH the dataset grant. `{{user_id}}` renders to the local brokered KC
# UUID when the user is auto-provisioned on first mesh request, so the
# viewer relation lands on the subject the per-dataset check looks up.

curl -sk -X POST "https://<TARGET_IP>/api/cluster/federations/<FEDERATION_ID>/users" \
  -H "Authorization: Bearer $REMOTE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "external_id": "<source-user-uuid>@<source-cluster-uuid>",
    "initial_tuples": [\
      {\
        "subject": "user:{{user_id}}",\
        "relation": "viewer",\
        "object": "dataset:<dataset-urn>"\
      }\
    ]
  }'

initial_tuples entries use namespaced "<namespace>:<id>" strings; {{user_id}} (local brokered UUID) and {{external_id}} (<source-user-uuid>@<source-cluster-uuid>) are substituted at provision time. To add a grant for an already-provisioned brokered user, re-POST the allowlist entry — seeding is idempotent.

Mesh Proxy Endpoint Reference

All federated operations use the mesh proxy at /api/mesh/{cluster_selector}/:

Timeouts

Headers

The mesh proxy adds these headers to cross-cluster requests: