Version: 1.0.1 (Latest) This guide walks through creating a federation between two Kamiwaza clusters and verifying cross-cluster operations. ## Prerequisites Before pairing, ensure each cluster has: 1. **Istio routing enabled**: `KAMIWAZA_ROUTING_PROVIDER=istio` 2. **STRICT mTLS**: `PeerAuthentication` in the `kamiwaza` namespace 3. **Node IP in gateway cert SANs**: The mesh proxy connects by IP; the TLS cert must include it ### Add Node IP to Gateway Certificate Each cluster's Istio gateway cert must include its node IP as a Subject Alternative Name (SAN). Without this, TLS verification fails when the mesh proxy connects by IP. **Option A: Via overrides.yaml (persistent)** Add to `cluster/values/overrides.yaml`: ```yaml global: istio: ingress: ipAddresses: - "192.168.50.168" # This cluster's node IP extraDnsNames: - "studio-1.example.com" # Optional hostname ``` Then sync: `helmfile -f cluster/helmfile.yaml.gotmpl sync -l name=kamiwaza` **Option B: Direct kubectl patch (quick)** ```bash kubectl patch certificate kamiwaza-gateway-tls -n istio-system --type merge -p '{ "spec": { "ipAddresses": ["192.168.50.168"], "dnsNames": ["localhost","kamiwaza.test","*.kamiwaza.test","studio-1.example.com"] } }' # Force re-issue kubectl delete secret kamiwaza-tls -n istio-system ``` **Verify:** ```bash kubectl get secret kamiwaza-tls -n istio-system -o jsonpath='{.data.tls\.crt}' \ | base64 -d | openssl x509 -noout -text | grep -A5 "Subject Alternative Name" # Should include: IP Address:192.168.50.168 ``` On k0s-lima clusters, `install-dev.sh` auto-detects the node IP and writes it to `overrides.yaml`. ## Step 1: Create Receiver on Remote Cluster The receiver creates a WAITING federation record. Authenticate to the remote cluster and create it: ```bash # Login to remote cluster REMOTE_TOKEN=$(curl -sk "https:///api/auth/token" -X POST \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'username=admin%40kamiwaza.localhost&password=&grant_type=password' \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])') # Create WAITING receiver curl -sk -X POST "https:///api/cluster/federations" \ -H "Authorization: Bearer $REMOTE_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "remote_cluster_name": "", "remote_ips": [{"ip": "", "primary": true}], "preshared_key": "", "callback_hostname": "", "role": "receiver" }' ``` Use the same `preshared_key` on both clusters. Set `callback_hostname` to the cluster's bridged VM IP — without it, the internal pod hostname is used, which is unreachable from the other cluster. ## Step 2: Create Initiator on Local Cluster ```bash # Login to local cluster LOCAL_TOKEN=$(curl -sk "https:///api/auth/token" -X POST \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'username=admin%40kamiwaza.localhost&password=&grant_type=password' \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])') # Create PAIRING initiator curl -sk -X POST "https:///api/cluster/federations" \ -H "Authorization: Bearer $LOCAL_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "remote_cluster_name": "", "remote_ips": [{"ip": "", "primary": true}], "preshared_key": "", "callback_hostname": "" }' # Note the federation ID from the response ``` ## Step 3: Pair ```bash curl -sk -X POST "https:///api/cluster/federations//pair" \ -H "Authorization: Bearer $LOCAL_TOKEN" # Response: status should be "PAIRED" ``` On success, both clusters transition to `PAIRED` and ReBAC relations are seeded: - **Source cluster**: `federation:{id}:operator` for the admin who paired - **Target cluster**: `cluster_jobs:__all__:executor` for the remote admin ## Step 4: Store Remote CA Certificate The mesh proxy needs to trust the remote cluster's TLS certificate. Fetch the remote cluster's root CA and store it in the federation record: ```bash # Get remote cluster's root CA cert REMOTE_CA=$(kubectl get secret root-ca -n kamiwaza -o jsonpath='{.data.ca\.crt}' | base64 -d) # Store in federation record kubectl exec core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza -c \ "UPDATE cluster_federations SET remote_ca_cert = '$(echo "$REMOTE_CA" | sed "s/'/''/g")' WHERE id = '';" ``` CA cert exchange will be automated in the pairing handshake in a future release. For now, manual storage is required. ## Step 5: Verify ```bash # Query remote catalog through mesh proxy curl -sk "https:///api/mesh//api/catalog/datasets/" \ -H "Authorization: Bearer $LOCAL_TOKEN" # Should return dataset list from the remote cluster ``` ## Disconnecting ```bash curl -sk -X POST "https:///api/cluster/federations//disconnect" \ -H "Authorization: Bearer $TOKEN" ``` Disconnecting immediately blocks new mesh proxy requests. Running operations on the remote cluster are not cancelled. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | 502 `mesh_proxy_bad_gateway` | TLS verification failed | Store remote CA cert (Step 4) or add IP to gateway cert SANs | | 401 on remote cluster | Mesh HMAC verification failed | Check PSK matches; check Istio `includeRequestHeadersInCheck` includes `x-kz-mesh-*` headers | | 403 `rebac_denied` on source | User lacks `federation:operator` relation | Re-pair the federation (seeding happens at pair time) | | 403 `namespace_unsupported` | `federation` namespace not registered | Ensure `authz/constants.py` includes `federation` in `ALLOWED_OBJECT_NAMESPACES` | | 400 `missing_object_id` | Cluster selector did not resolve to a PAIRED federation | Check federation status; selector must match a PAIRED federation by name, UUID, or prefix | | 403 `not_authorized_to_probe_cluster` | Mesh-origin capabilities probe lacks a `cluster:` viewer grant | Grant the federated subject the `cluster:` viewer relation explicitly before probing | | 307 redirect | Missing trailing slash | Add `/` to the API path |