This is documentation for Kamiwaza **1.0.1**, which is no longer actively maintained. For the current GA release, see [**1.0.1**](https://docs.kamiwaza.ai/).

Version: 1.0.1

## Federation Management
Manage cluster federation pairing and cluster metadata.

### List Federations
```text
GET /api/cluster/federations
```
Returns all federations the current user has operator or viewer access to.

**Response:**
```json
[
  {
    "id": "uuid",
    "remote_cluster_name": "string",
    "remote_ips": [{"ip": "string", "primary": true}],
    "callback_hostname": "string",
    "local_cluster_id": "uuid",
    "remote_cluster_id": "uuid",
    "status": "PAIRED | PAIRING | FAILED",
    "last_ping": "timestamp",
    "created_at": "timestamp"
  }
]
```

### Pair a Federation
```text
POST /api/cluster/federations/pair
```
Initiates a pairing handshake with a remote cluster. Exchanges CA certificates and pre-shared HMAC keys. Requires admin role.

**Request:**
```json
{
  "remote_cluster_name": "string",
  "remote_ips": ["192.168.50.168"],
  "local_ca_cert": "-----BEGIN CERTIFICATE-----\n..."
}
```

### Unpair a Federation
```text
DELETE /api/cluster/federations/{federation_id}
```  
Tears down the federation. Removes ReBAC grants and cleans up the pre-shared key.

### Ping a Federation
```text
GET /api/cluster/federations/{federation_id}/ping
```
Tests end-to-end connectivity through the mesh proxy. Returns reachability status without requiring authentication to succeed on the remote side.

## Mesh Proxy
The mesh proxy forwards requests to remote federated clusters. Every request is HMAC-signed and ReBAC-gated.

### Proxy Path Pattern
```text
{METHOD} /api/mesh/{federation_name}/{remote_path}
```
The `{federation_name}` is the `remote_cluster_name` from the federation record. The `{remote_path}` is the path on the remote cluster (without the `/api` prefix — it's re-added by the proxy).

**Authorization:** The caller must have `operator` relation on the federation (seeded automatically when the federation is paired by an admin; can be granted to other users via the ReBAC API).

**Request headers forwarded upstream:**

| Header | Source | Purpose |
| --- | --- | --- |
| `X-KZ-Mesh-Source-Cluster-Id` | Local cluster ID | Identifies the originating cluster |
| `X-KZ-Mesh-User-Id` | Local user's `sub` claim | Remote identity resolution |
| `X-KZ-Mesh-User-Roles` | Local user's roles (CSV) | Remote role-based checks |
| `X-KZ-Mesh-Route` | `{method} {path}` | Bound into the HMAC signature |
| `X-KZ-Mesh-Signature` | HMAC-SHA256 | Verified on the remote cluster |
| `X-KZ-Mesh-Signature-Ts` | Unix timestamp | Replay protection (5-minute window) |
| `X-KZ-Mesh-Correlation-Id` | Per-request UUID | Observability tracing |
| `X-KZ-Mesh-User-Attributes` | `X-User-Attributes` from source | Passed through to attribute gates |

### Common Proxied Endpoints

| Path | Purpose |
| --- | --- |
| `GET /api/mesh/{fed}/catalog/datasets/` | List remote datasets |
| `GET /api/mesh/{fed}/serving/deployments` | List remote models |
| `POST /api/mesh/{fed}/retrieval/jobs` | Run a retrieval job on the remote cluster |
| `GET /api/mesh/{fed}/retrieval/jobs/{id}/stream` | SSE stream of a remote retrieval |
| `POST /api/mesh/{fed}/cluster/jobs/run` | Submit and run a Ray job on the remote cluster |

### Error Responses

| Status | Condition |
| --- | --- |
| `401` | Local auth failed (invalid JWT / PAT) |
| `403` | Caller lacks operator on the federation or remote ReBAC blocks the operation |
| `503` | Remote cluster unreachable or HMAC verification failed on the remote side |
| `504` | Remote request exceeded the proxy timeout |

## Job Submission
Submit and manage Ray jobs on local or remote clusters. For the full lifecycle narrative with examples, see [Job Submission](https://docs.kamiwaza.ai/1.0.1/federation/job-submission).

### Submit Asynchronously
```text
POST /api/cluster/jobs/submit
```
Submits a job to Ray and returns immediately. Poll `/status` or `/result` to track it.

**Request:**
```json
{
  "cluster_selector": "local" | "federation_name",
  "entrypoint": "python analysis.py",
  "runtime_env": {
    "env_vars": {"KEY": "value"},
    "working_dir": "s3://..."
  },
  "metadata": {"label": "value"},
  "timeout_seconds": 300
}
```

**Response:**
```json
{
  "id": "uuid",
  "ray_job_id": "string",
  "status": "PENDING",
  "cluster_selector": "string",
  "submitted_at": "timestamp"
}
```

### Run Synchronously
```text
POST /api/cluster/jobs/run
```
Submits the job, polls until completion (or timeout), extracts the result marker, and returns everything in one response.

**Response:**
```json
{
  "id": "uuid",
  "status": "SUCCEEDED" | "FAILED" | "TIMEOUT" | "CANCELLED",
  "result": { ... },
  "duration_seconds": 3.1,
  "error_message": "string | null"
}
```

### Get Status
```text
GET /api/cluster/jobs/{job_id}/status
```

### Get Result
```text
GET /api/cluster/jobs/{job_id}/result
```
Returns the structured result extracted from the job's log marker (`KZ_MESH_RUN_ON_JSON::{...}`).

### Get Logs
```text
GET /api/cluster/jobs/{job_id}/logs
```
Returns Ray stdout/stderr for the job.

### Cancel
```text
POST /api/cluster/jobs/{job_id}/cancel
```
Signals Ray to cancel a running job.

## ReBAC Relations
Fine-grained authorization. All mesh and federation operations go through ReBAC checks.

### Grant a Relation
```text
POST /api/auth/tuples
```
**Request:**
```json
{
  "subject": {"namespace": "user", "id": "<uuid>"},
  "relation": "operator" | "viewer" | "owner" | "executor",
  "object": {
    "namespace": "federation" | "dataset" | "model" | "cluster_jobs",
    "id": "string"
  }
}
```
This grants a relation to a **local** user — one with an account on the cluster you call. It does **not** work for brokered mesh users (see the note below).

### Common Grant Patterns

| Scenario | Namespace | Relation | Notes |
| --- | --- | --- | --- |
| User can use a federation | `federation` | `operator` | Required to call `/api/mesh/{fed}/*`; granted to a local user on the **source** cluster |
| Local user can query a dataset | `dataset` | `viewer` | For native (non-mesh) retrieval on this cluster |
| User can submit jobs | `cluster_jobs` | `executor` | Object id is the constant `"__all__"` |
| User can own a dataset | `dataset` | `owner` | Can write and manage |

## Attribute Headers
User attributes flow through ext-authz from JWT claims to domain gates on the retrieval service.

### X-User-Attributes
Generic JSON header carrying all custom Keycloak user attributes. Set by ext-authz on the gateway when the JWT has custom claims (e.g. `clearance`, `country`, `department`).

**Example:**
```text
X-User-Attributes: {"clearance":"S","country":"GBR"}
```
Forwarded across the mesh as `X-KZ-Mesh-User-Attributes`. Attribute gates on the retrieval service read this header to filter records per-user.