Operations & Troubleshooting | Kamiwaza Docs

Version: 1.0.1 (Latest)

1. Federation Setup

Prerequisites

Before attempting to pair two clusters, verify:

1. Both clusters run Istio with KAMIWAZA_ROUTING_PROVIDER=istio
2. STRICT mTLS is configured (PeerAuthentication in the kamiwaza namespace)
3. ReBAC is enabled on both clusters
4. Network connectivity exists between clusters on port 443
5. Gateway certs include node IPs in Subject Alternative Names (SANs)
6. NTP is synchronized on both clusters (clock skew must be under 300 seconds)

Step-by-Step Pairing

1. Authenticate to both clusters:

# Remote cluster (the receiver)

REMOTE_TOKEN=$(curl -sk "https://192.168.50.13/api/auth/token" -X POST \

-H 'Content-Type: application/x-www-form-urlencoded' \

-d 'username=admin%40kamiwaza.localhost&password=kamiwaza&grant_type=password' \

| python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

# Local cluster (the initiator)

LOCAL_TOKEN=$(curl -sk "https://kamiwaza.test/api/auth/token" -X POST \

-H 'Content-Type: application/x-www-form-urlencoded' \

-d 'username=admin%40kamiwaza.localhost&password=kamiwaza&grant_type=password' \

| python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

2. Create WAITING receiver on the remote cluster:

curl -sk -X POST "https://192.168.50.13/api/cluster/federations" \

-H "Authorization: Bearer $REMOTE_TOKEN" \

-H "Content-Type: application/json" \

-d '{

"remote_cluster_name": "dev-laptop",

"remote_ips": [{"ip": "192.168.50.168", "primary": true}],

"preshared_key": "my-strong-shared-secret",

"callback_hostname": "192.168.50.13",

"role": "receiver"

}'

3. Create PAIRING initiator on the local cluster:

FEDERATION_RESPONSE=$(curl -sk -X POST "https://kamiwaza.test/api/cluster/federations" \

-H "Authorization: Bearer $LOCAL_TOKEN" \

-H "Content-Type: application/json" \

-d '{

"remote_cluster_name": "studio-1",

"remote_ips": [{"ip": "192.168.50.13", "primary": true}],

"preshared_key": "my-strong-shared-secret",

"callback_hostname": "192.168.50.168"

}')

FEDERATION_ID=$(echo "$FEDERATION_RESPONSE" | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')

echo "Federation ID: $FEDERATION_ID"

4. Initiate pairing:

curl -sk -X POST "https://kamiwaza.test/api/cluster/federations/$FEDERATION_ID/pair" \

-H "Authorization: Bearer $LOCAL_TOKEN"

5. Store remote CA certificate:

# Fetch the remote cluster's root CA (run on the remote cluster or via SSH)

REMOTE_CA=$(kubectl get secret root-ca -n kamiwaza -o jsonpath='{.data.ca\.crt}' | base64 -d)

# Store in federation record on the local cluster

kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza -c \

"UPDATE cluster_federations SET remote_ca_cert = '$(echo "$REMOTE_CA" | sed "s/'/''/g")' WHERE id = '$FEDERATION_ID';"

Repeat the CA cert step in the other direction (store the local CA on the remote cluster's federation record).

Verifying Pairing Succeeded

# Check federation status

curl -sk "https://kamiwaza.test/api/cluster/federations/$FEDERATION_ID" \

-H "Authorization: Bearer $LOCAL_TOKEN" | python3 -m json.tool

# Expected: "status": "PAIRED"

# Test cross-cluster connectivity by listing remote datasets

curl -sk "https://kamiwaza.test/api/mesh/studio-1/api/catalog/datasets/" \

-H "Authorization: Bearer $LOCAL_TOKEN"

# Test remote model listing

curl -sk "https://kamiwaza.test/api/mesh/studio-1/api/serving/deployments" \

-H "Authorization: Bearer $LOCAL_TOKEN"

2. Mesh Proxy Troubleshooting

HMAC Signature Failures (403)

Symptoms: 403 response on mesh proxy requests. No rebac_denied detail in the response body -- the request is rejected before reaching the endpoint.

Diagnosis:

1. Check PSK match -- The preshared_key must be identical on both sides of the federation:

# On the local cluster

kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza -c \

"SELECT id, remote_cluster_name, preshared_key FROM cluster_federations WHERE status = 'PAIRED';"

Compare with the same query on the remote cluster. The PSK values must match exactly.

2. Check clock skew -- The HMAC signature includes a timestamp. The receiving cluster rejects signatures older than 300 seconds (configurable via auth_forward_header_signature_ttl_seconds):

# Compare times on both clusters

date -u  # Run on both clusters and compare

3. Check that Istio forwards mesh headers -- The ext-authz configuration must include all X-KZ-Mesh-* headers in includeRequestHeadersInCheck:

kubectl get configmap istio -n istio-system -o yaml | grep -A 20 "extensionProviders"

# Verify X-KZ-Mesh-Source-Cluster-Id, X-KZ-Mesh-Signature, etc. are listed

Resolution:

• If PSKs do not match: disconnect and re-pair the federation with matching keys.
• If clock skew exceeds 300 seconds: synchronize NTP on both clusters.
• If headers are missing from ext-authz config: update the Istio mesh config and restart the ext-authz pod.

ReBAC Denied (403 rebac_denied)

Symptoms: 403 with rebac_denied detail in the response body. The HMAC signature was valid (the request reached the endpoint), but the user lacks authorization on the target cluster.

Diagnosis:

1. Check the authorization_relations table on the target cluster. Note the relevant subject_id is the local brokered KC UUID (auto-provisioned on first mesh request), not the source cluster's user UUID — query by relation/object, or list the brokered user's local UUID first via GET /api/cluster/federations/{id}/users:

kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza -c \

"SELECT subject_namespace, subject_id, relation, object_namespace, object_id

FROM authorization_relations

WHERE object_namespace = 'dataset' AND object_id = '<dataset-urn>';"

2. For catalog/retrieval operations, the remote user needs a viewer relation on the dataset namespace for each dataset they access.
3. Admin bypass is suppressed for mesh requests. Even if the remote user has an admin role on their home cluster, they still need explicit ReBAC grants on the target cluster for per-resource operations.

Resolution:

Seed the per-dataset grant through the brokered-user allowlist's initial_tuples field. Do not use /api/auth/tuples for a brokered user: that user has no local account on the target cluster until their first mesh request, when brokering auto-provisions a local Keycloak user with a freshly-minted UUID. The check authorizes against that local UUID, so a tuple written against the source UUID returns 204 but never matches (access stays 404). initial_tuples resolves the {{user_id}} placeholder to the local UUID at provision time.

# Grant viewer on a dataset to a brokered user, via the allowlist. Re-POSTing

# the entry for an already-provisioned user is idempotent.

curl -sk -X POST "https://<TARGET_IP>/api/cluster/federations/<FEDERATION_ID>/users" \

-H "Authorization: Bearer $REMOTE_ADMIN_TOKEN" \

-H "Content-Type: application/json" \
  -d '{

"external_id": "<source-user-uuid>@<source-cluster-uuid>",

"initial_tuples": [\
\
      {"subject": "user:{{user_id}}", "relation": "viewer", "object": "dataset:<dataset-urn>"}\
\
    ]

}'

For federation operator access on the source cluster (needed to use the mesh proxy at all) — this is a local user on its home cluster, so /api/auth/tuples is the correct path here:

# Grant operator on federation namespace (normally seeded during pairing)

curl -sk -X POST "https://kamiwaza.test/api/auth/tuples" \

-H "Authorization: Bearer $LOCAL_ADMIN_TOKEN" \

-H "Content-Type: application/json" \
  -d '{

"subject": {"namespace": "user", "id": "<user-uuid>"},

"relation": "operator",

"object": {"namespace": "federation", "id": "<federation-id>"}

}'

CA Trust Errors (TLS)

Symptoms:502 mesh_proxy_bad_gateway response. Connection refused or TLS handshake failure in the core service logs.

Diagnosis:

1. Check that remote_ca_cert is populated in the cluster_federations table:

kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza -c \

"SELECT id, remote_cluster_name,

CASE WHEN remote_ca_cert IS NOT NULL THEN 'SET' ELSE 'NULL' END AS ca_cert_status

FROM cluster_federations WHERE status = 'PAIRED';"

2. Verify the remote cluster's gateway cert includes its node IP in SANs:

# Run on the remote cluster

kubectl get secret kamiwaza-tls -n istio-system -o jsonpath='{.data.tls\.crt}' \
     | base64 -d | openssl x509 -noout -text | grep -A5 "Subject Alternative Name"

3. Test TLS connectivity manually:

openssl s_client -connect 192.168.50.13:443 -CAfile /tmp/remote-ca.pem

Resolution:

• If remote_ca_cert is NULL: re-pair the federation, or manually inject the CA cert (see setup step 5).
• If the remote cert does not include the node IP: update the gateway certificate SANs.

Trailing Slash Redirects (307 -> auth loss)

Symptoms: 307 redirect followed by 401 or 403 on the redirected request. The HMAC signature headers are lost during the redirect because the mesh proxy has follow_redirects=False.

Diagnosis:

Check the logs for 307 status codes:

kubectl logs deployment/core-scheduler -n kamiwaza | grep "307"

The HMAC signature binds to the specific request path. If the server redirects (e.g., /api/catalog/datasets to /api/catalog/datasets/), the signature becomes invalid for the new path.

Resolution:

Ensure API paths in mesh requests use the canonical form (with or without trailing slash as the endpoint expects). Most list endpoints expect a trailing slash:

# Correct

curl -sk "https://kamiwaza.test/api/mesh/studio-1/api/catalog/datasets/" ...

# May cause 307 redirect

curl -sk "https://kamiwaza.test/api/mesh/studio-1/api/catalog/datasets" ...

Clock Skew

Symptoms: Intermittent 403 errors on mesh requests. Requests succeed sometimes and fail at other times, especially near the TTL boundary.

Diagnosis:

# Check the time on both clusters

ssh cluster-a "date -u '+%s'"

ssh cluster-b "date -u '+%s'"

# Difference must be < 300 seconds for mesh, < 60 seconds for pairing

The mesh signature TTL is 300 seconds by default (auth_forward_header_signature_ttl_seconds). The allowed future skew is 60 seconds (auth_forward_header_signature_allowed_future_skew_seconds).

Resolution:

Synchronize NTP on both clusters:

# On each cluster node

sudo timedatectl set-ntp true

timedatectl status

3. Job Monitoring

Job States

Checking Job Status

Local job:

curl -sk "https://kamiwaza.test/api/cluster/jobs/<job-id>/status" \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool

Federated job (submitted via mesh):

curl -sk "https://kamiwaza.test/api/mesh/studio-1/cluster/jobs/<job-id>/status" \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool

Checking Job Logs

# Local job logs

curl -sk "https://kamiwaza.test/api/cluster/jobs/<job-id>/logs" \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool

# Federated job logs

curl -sk "https://kamiwaza.test/api/mesh/studio-1/cluster/jobs/<job-id>/logs" \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool

Result Extraction

Jobs return structured results via a log marker protocol. The job code prints a marker to stdout:

import json

result = {"accuracy": 0.95, "model": "v2"}

print(f"KZ_MESH_RUN_ON_JSON::{{json.dumps(result)}}")

The platform scans the job's Ray logs (reverse, last match wins) for KZ_MESH_RUN_ON_JSON:: and parses the JSON payload that follows.

Extract the result:

curl -sk "https://kamiwaza.test/api/cluster/jobs/<job-id>/result" \
  -H "Authorization: Bearer $TOKEN"

Error responses:

Timeout and Auto-Cancel

When timeout_seconds is set on job submission and the job is executed via the synchronous /run endpoint, the platform polls Ray until the timeout elapses. If the job is still running at that point:

1. The job is cancelled via the Ray Dashboard API
2. The timed_out flag is set to true in the federated_jobs table
3. The job status transitions to STOPPED

The timeout applies only during synchronous /run calls. Async /submit does not enforce timeouts automatically.

4. Database Queries

Useful psql queries for monitoring federation state. Connect via:

kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza

List all federations and their status

SELECT id, remote_cluster_name, status, last_ping,

CASE WHEN remote_ca_cert IS NOT NULL THEN 'yes' ELSE 'NO' END AS has_ca_cert,

created_at

FROM cluster_federations

ORDER BY created_at DESC;

List all federated jobs

SELECT id, status, source, source_cluster_id, source_cluster_name,

user_id, timed_out, submitted_at, ended_at

FROM federated_jobs

ORDER BY submitted_at DESC

LIMIT 20;

List mesh-originated jobs only

SELECT id, status, source_cluster_name, user_id,

entrypoint, timed_out, error_type

FROM federated_jobs

WHERE source = 'mesh'

ORDER BY submitted_at DESC;

Check ReBAC relations for federation

SELECT subject_namespace, subject_id, relation, object_namespace, object_id

FROM authorization_relations

WHERE object_namespace = 'federation'

ORDER BY created_at DESC;

Find failed jobs with error details

SELECT id, source_cluster_name, entrypoint,

error_type, error_message, ended_at

FROM federated_jobs

WHERE status = 'FAILED'

ORDER BY ended_at DESC

LIMIT 10;

Check for timed-out jobs

SELECT id, source, source_cluster_name, entrypoint,

timeout_seconds, submitted_at, ended_at

FROM federated_jobs

WHERE timed_out = true

ORDER BY ended_at DESC;

5. Useful Headers for Debugging

All cross-cluster mesh requests include these headers. They are set by the mesh proxy on the source cluster and verified by ext-authz on the target cluster.

Signature Payload

The HMAC signature binds to these fields (all lowercased in the canonical form):

Changing any of these fields after signing invalidates the signature. This prevents replay attacks across different endpoints or methods.