Federation Setup | Kamiwaza Docs

This is documentation for Kamiwaza 1.0.0, which is no longer actively maintained. For the current GA release, see 1.0.1.

Version: 1.0.0

Prerequisites

Before pairing, ensure each cluster has:

1. Istio routing enabled: KAMIWAZA_ROUTING_PROVIDER=istio
2. STRICT mTLS: PeerAuthentication in the kamiwaza namespace
3. Node IP in gateway cert SANs: The mesh proxy connects by IP; the TLS cert must include it

Add Node IP to Gateway Certificate

Each cluster's Istio gateway cert must include its node IP as a Subject Alternative Name (SAN). Without this, TLS verification fails when the mesh proxy connects by IP.

Option A: Via overrides.yaml (persistent)

Add to cluster/values/overrides.yaml:

global:
  istio:
    ingress:
      ipAddresses:
        - "192.168.50.168"       # This cluster's node IP
      extraDnsNames:
        - "studio-1.example.com" # Optional hostname

Then sync: helmfile -f cluster/helmfile.yaml.gotmpl sync -l name=kamiwaza

Option B: Direct kubectl patch (quick)

kubectl patch certificate kamiwaza-gateway-tls -n istio-system --type merge -p '{
  "spec": {
    "ipAddresses": ["192.168.50.168"],
    "dnsNames": ["localhost","kamiwaza.test","*.kamiwaza.test","studio-1.example.com"]
  }
}'

# Force re-issue

kubectl delete secret kamiwaza-tls -n istio-system

Verify:

kubectl get secret kamiwaza-tls -n istio-system -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep -A5 "Subject Alternative Name"

# Should include: IP Address:192.168.50.168

note On k0s-lima clusters, install-dev.sh auto-detects the node IP and writes it to overrides.yaml.

Step 1: Create Receiver on Remote Cluster

The receiver creates a WAITING federation record. Authenticate to the remote cluster and create it:

# Login to remote cluster

REMOTE_TOKEN=$(curl -sk "https://<REMOTE_IP>/api/auth/token" -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=admin%40kamiwaza.localhost&password=<PASSWORD>&grant_type=password' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

# Create WAITING receiver

curl -sk -X POST "https://<REMOTE_IP>/api/cluster/federations" \
  -H "Authorization: Bearer $REMOTE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "remote_cluster_name": "<local-cluster-name>",
    "remote_ips": [{"ip": "<LOCAL_IP>", "primary": true}],
    "preshared_key": "<shared-secret>",
    "callback_hostname": "<REMOTE_IP>",
    "role": "receiver"
  }'

important Use the same preshared_key on both clusters. Set callback_hostname to the cluster's bridged VM IP — without it, the internal pod hostname is used, which is unreachable from the other cluster.

Step 2: Create Initiator on Local Cluster

# Login to local cluster

LOCAL_TOKEN=$(curl -sk "https://<LOCAL_IP>/api/auth/token" -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=admin%40kamiwaza.localhost&password=<PASSWORD>&grant_type=password' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')

# Create PAIRING initiator

curl -sk -X POST "https://<LOCAL_IP>/api/cluster/federations" \
  -H "Authorization: Bearer $LOCAL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "remote_cluster_name": "<remote-cluster-name>",
    "remote_ips": [{"ip": "<REMOTE_IP>", "primary": true}],
    "preshared_key": "<shared-secret>",
    "callback_hostname": "<LOCAL_IP>"
  }'

# Note the federation ID from the response

Step 3: Pair

curl -sk -X POST "https://<LOCAL_IP>/api/cluster/federations/<FEDERATION_ID>/pair" \
  -H "Authorization: Bearer $LOCAL_TOKEN"

# Response: status should be "PAIRED"

On success, both clusters transition to PAIRED and ReBAC relations are seeded:

• Source cluster: federation:{id}:operator for the admin who paired
• Target cluster: cluster_jobs:__all__:executor for the remote admin

Step 4: Store Remote CA Certificate

The mesh proxy needs to trust the remote cluster's TLS certificate. Fetch the remote cluster's root CA and store it in the federation record:

# Get remote cluster's root CA cert

REMOTE_CA=$(kubectl get secret root-ca -n kamiwaza -o jsonpath='{.data.ca\.crt}' | base64 -d)

# On the remote cluster, or via SSH

# Store in federation record

kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza -c \
  "UPDATE cluster_federations SET remote_ca_cert = '$(echo "$REMOTE_CA" | sed "s/'/''/g")' WHERE id = '<FEDERATION_ID>';"

note CA cert exchange will be automated in the pairing handshake in a future release. For now, manual storage is required.

Step 5: Verify

# Query remote catalog through mesh proxy

curl -sk "https://<LOCAL_IP>/api/mesh/<remote-cluster-name>/api/catalog/datasets/" \
  -H "Authorization: Bearer $LOCAL_TOKEN"

# Should return dataset list from the remote cluster

# List remote models

curl -sk "https://<LOCAL_IP>/api/mesh/<remote-cluster-name>/api/serving/deployments" \
  -H "Authorization: Bearer $LOCAL_TOKEN"

# Should return deployment list from the remote cluster

Disconnecting

curl -sk -X POST "https://<IP>/api/cluster/federations/<FEDERATION_ID>/disconnect" \
  -H "Authorization: Bearer $TOKEN"

Disconnecting immediately blocks new mesh proxy requests. Running operations on the remote cluster are not cancelled.

Troubleshooting