Federated Retrieval | Kamiwaza Docs

Version: 1.0.1 (Latest)

Federated Retrieval

Federated retrieval lets you discover and query data on remote clusters without the data leaving its origin. The mesh proxy routes your requests transparently — the remote cluster's catalog and retrieval services process them as if they were local.

Prerequisites

• A paired federation between the clusters
• Remote CA cert stored in the federation record
• The mesh user must have per-dataset ReBAC grants on the target cluster (admin users with extauthz trust are exempt)

Discover Remote Datasets

# List datasets on the remote cluster

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/catalog/datasets/" \
  -H "Authorization: Bearer $TOKEN"

Response: array of dataset metadata (URNs, names, schemas) from the remote cluster's catalog.

Retrieve Remote Data

# Create a retrieval job on the remote cluster

curl -sk -X POST "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_urn": "urn:li:dataset:(urn:li:dataPlatform:postgres,nps_verbatims,PROD)",
    "columns": ["customer_id", "verbatim", "score"],
    "filters": {"quarter": "Q1-2026"},
    "transport": "inline"
  }'

The retrieval service on the remote cluster accesses its local data adapters (S3, Postgres, Hive, etc.) and returns the results through the mesh proxy. Only the query results cross the network — raw data stays on the remote cluster.

Streaming Retrieval (SSE)

For large result sets, use the streaming transport:

# Submit job, then stream results

JOB_ID=$(curl -sk -X POST "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataset_urn": "...", "transport": "sse"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["job_id"])')

# Stream results via SSE

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs/$JOB_ID/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

SSE events stream through the mesh proxy transparently.

List Remote Models

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/serving/deployments" \
  -H "Authorization: Bearer $TOKEN"

Returns deployment metadata: model names, engine types, status, and access paths.

Run Remote Inference

# Chat completion on a remote model

curl -sk -X POST \
  "https://kamiwaza.test/api/mesh/<remote-cluster>/runtime/models/<deployment-id>/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-4B",
    "messages": [{"role": "user", "content": "Summarize the NPS themes"}]
  }'

The inference request routes through the mesh proxy to the remote cluster's Ray Serve deployment. The response streams back.

Access Control

Admin Users

On Istio clusters with the extauthz trust model, admin users bypass per-resource ReBAC checks on the target cluster (standard admin bypass). No additional seeding is needed.

Non-Admin Users

Non-admin users require explicit per-dataset ReBAC grants on the target cluster. The admin bypass is suppressed for mesh-originated requests — federated callers must have explicit viewer access to each dataset they query or browse. The catalog listing endpoint filters results so federated users only see datasets they have grants for.

Grant access at federation-pairing time through the brokered-user allowlist's initial_tuples field — not through /api/auth/tuples.

# On the TARGET cluster, add the source user to the federation allowlist
# WITH the dataset grant. `{{user_id}}` renders to the local brokered KC
# UUID when the user is auto-provisioned on first mesh request, so the
# viewer relation lands on the subject the per-dataset check looks up.

curl -sk -X POST "https://<TARGET_IP>/api/cluster/federations/<FEDERATION_ID>/users" \
  -H "Authorization: Bearer $REMOTE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "external_id": "<source-user-uuid>@<source-cluster-uuid>",
    "initial_tuples": [\
      {\
        "subject": "user:{{user_id}}",\
        "relation": "viewer",\
        "object": "dataset:<dataset-urn>"\
      }\
    ]
  }'

initial_tuples entries use namespaced "<namespace>:<id>" strings; {{user_id}} (local brokered UUID) and {{external_id}} (<source-user-uuid>@<source-cluster-uuid>) are substituted at provision time. To add a grant for an already-provisioned brokered user, re-POST the allowlist entry — seeding is idempotent.

Mesh Proxy Endpoint Reference

All federated operations use the mesh proxy at /api/mesh/{cluster_selector}/:

Timeouts

Headers

The mesh proxy adds these headers to cross-cluster requests: