# Documentation for Kamiwaza 1.0.0

This is documentation for Kamiwaza **1.0.0**, which is no longer actively maintained. For the current GA release, see [**1.0.1**](https://docs.kamiwaza.ai/).

Version: 1.0.0

## Overview

Kamiwaza cluster federation enables cross-cluster operations between paired Kamiwaza clusters. A federated cluster can query remote data, list remote models, and execute compute jobs on its partner — all through a secured mesh proxy with HMAC-signed trust and ReBAC authorization.

## Key Concepts

**Mesh Proxy** — Routes API calls from one cluster to another. The proxy strips the caller's authentication, signs the request with the federation's pre-shared key (PSK) via HMAC-SHA256, and forwards to the remote cluster's Istio ingress gateway. The response streams back transparently.

**Federation Trust** — Each federation pair shares a pre-shared key. All cross-cluster requests are HMAC-signed with this key. The receiving cluster verifies the signature before processing the request. Trust is per-federation, not per-cluster.

**ReBAC Gating** — Authorization is enforced on both sides of the federation link:

- **Source cluster**: The calling user must have the `operator` relation on the `federation` namespace for the target federation
- **Target cluster**: The mesh-originated identity is evaluated against standard per-resource ReBAC guards (catalog, retrieval, jobs)

**Compute Ships to Data** — Raw data never leaves the originating cluster. Only structured results (API responses, job outputs) cross the federation boundary.

## What You Can Do

| Operation                     | How                                         | What Crosses the Network                     |
|-------------------------------|--------------------------------------------|----------------------------------------------|
| Query remote catalog          | `GET /api/mesh/{cluster}/api/catalog/datasets/` | Dataset metadata (names, schemas, URNs)     |
| Retrieve remote data          | `POST /api/mesh/{cluster}/api/retrieval/jobs` | Query results (rows matching your filter)    |
| List remote models            | `GET /api/mesh/{cluster}/api/serving/deployments` | Deployment metadata (model names, status)    |
| Run remote inference          | `POST /api/mesh/{cluster}/runtime/models/{id}/v1/chat/completions` | Inference results                            |
| Submit remote job             | `POST /api/mesh/{cluster}/cluster/jobs/run` | Structured job result (MVP+1)                |

## Architecture

```text
Cluster A (source)                          Cluster B (target)

┌─────────────────────┐                    ┌─────────────────────┐

│ User / Kaizen Agent  │                    │                     │

│   │                  │                    │                     │

│   ▼                  │                    │                     │

│ Mesh Proxy           │   HMAC-signed      │ Istio Gateway       │
│ /api/mesh/{cluster}/ │──────────────────▶│   │                 │
│   │ @guarded         │   HTTPS + mTLS     │   ▼                 │
│   │ (federation:     │                    │ ext-authz           │
│   │  operator)       │                    │   │ verify HMAC     │
│   │                  │                    │   │ mint identity   │
│   │                  │                    │   ▼                 │
│   │                  │                    │ @guarded endpoint   │
│   │                  │                    │   │ (standard ReBAC)│
│   │                  │                    │   ▼                 │
│   │◀─────────────────│◀──────────────────│ Service response    │
│   │ response         │                    │                     │
└─────────────────────┘                    └─────────────────────┘
```

## Prerequisites

- Two Kamiwaza clusters running on Istio with `KAMIWAZA_ROUTING_PROVIDER=istio`
- STRICT mTLS (`PeerAuthentication`) in both namespaces
- Network connectivity between clusters on port 443
- Each cluster's Istio gateway TLS cert must include its node IP in SANs

## Federation lifecycle at a glance

| Step                              | What                                                              | Where it's documented                                      |
|-----------------------------------|-------------------------------------------------------------------|-----------------------------------------------------------|
| Pair two clusters                 | Exchange PSK + CA certs; provision the federation record on both sides | [Setup](https://docs.kamiwaza.ai/1.0.0/federation/setup) |
| Allowlist federated users         | Decide which remote users can act on this cluster                 | [Setup](https://docs.kamiwaza.ai/1.0.0/federation/setup) |
| Query remote data                 | Federated retrieval over the mesh proxy                           | [Retrieval](https://docs.kamiwaza.ai/1.0.0/federation/retrieval) |
| Submit remote jobs                | Cluster-job execution with on-behalf-of (OBO) identity           | [Job Submission](https://docs.kamiwaza.ai/1.0.0/federation/job-submission) |
| Apply policy gates                | Cluster `ExecutionGate` and per-dataset `AttributeGate` bindings | [Execution Gates](https://docs.kamiwaza.ai/1.0.0/federation/execution-gates) |
| Install custom gates              | Ship custom policy code as Python packages                        | [Gate Packages](https://docs.kamiwaza.ai/1.0.0/federation/gate-packages) |
| Day-2 operations                  | Diagnose, cancel, revoke, audit                                   | [Operations](https://docs.kamiwaza.ai/1.0.0/federation/operations) |

## Next Steps

- [Federation Setup Guide](https://docs.kamiwaza.ai/1.0.0/federation/setup) — Create and pair federations
- [Federated Retrieval](https://docs.kamiwaza.ai/1.0.0/federation/retrieval) — Query remote data through the mesh proxy
- [Job Submission](https://docs.kamiwaza.ai/1.0.0/federation/job-submission) — Submit federated jobs with OBO identity
- [Execution Gates](https://docs.kamiwaza.ai/1.0.0/federation/execution-gates) — Cluster `ExecutionGate` + dataset `AttributeGate` policy bindings
- [Gate Packages](https://docs.kamiwaza.ai/1.0.0/federation/gate-packages) — Install custom gate classes as hash-pinned Python packages
- [Operations & Troubleshooting](https://docs.kamiwaza.ai/1.0.0/federation/operations) — Day-2 federation operations
- [API Reference](https://docs.kamiwaza.ai/1.0.0/federation/api-reference) — Endpoint catalog