This is documentation for Kamiwaza **0.9.0**, which is no longer actively maintained. For the current GA release, see [**1.0.1**](https://docs.kamiwaza.ai/).

Version: 0.9.0

## Platform assumptions

- Host meets the published system requirements. See [System Requirements](https://docs.kamiwaza.ai/0.9.0/installation/system_requirements) for the supported operating systems and hardware profiles.
- Installation created the managed systemd units or Docker Compose stack for Auth, Keycloak, and Traefik.
- You have shell access to restart those services and reach the Traefik gateway externally.

## Prerequisites

| Item | Notes |
| --- | --- |
| Keycloak 22.x+ | Administrative access to create a realm and confidential client. |
| Kamiwaza platform install | Current RPM build (or the equivalent compose stack). |
| Shell access | Ability to edit `env.sh`, run helper scripts, and restart Auth/Keycloak/Traefik. |
| TLS assets | Certificates for the public gateway hostname (self-signed is fine for labs). |

Before starting, stop the Auth, Keycloak, and Traefik services (docker compose or systemd) so configuration changes and realm imports apply cleanly. You will restart them later in this guide.

## Configure environment

Update `env.sh` (or export the variables directly) with the ReBAC settings below. Replace the placeholders with values that match your environment.

```bash
# Auth gateway ↔ Keycloak

export AUTH_GATEWAY_OIDC_ENABLED=true

export AUTH_GATEWAY_KEYCLOAK_URL=https://<keycloak-host>

export AUTH_GATEWAY_KEYCLOAK_REALM=<realm>

export AUTH_GATEWAY_KEYCLOAK_CLIENT_ID=kamiwaza-platform

export AUTH_CALLBACK_URL=https://<gateway-host>/api/auth/callback

# Optional: only set if the Keycloak client enforces confidential secret auth

# export AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET=<client-secret>

export AUTH_GATEWAY_PUBLIC_URL=https://<gateway-host>

export AUTH_GATEWAY_COOKIE_DOMAIN=<gateway-host>

# ForwardAuth secret shared with Traefik

export AUTH_FORWARD_AUTH_STRICT=true

export AUTH_FORWARD_HEADER_SECRET=<64-hex-secret>

# ReBAC session store

export AUTH_REBAC_ENABLED=true

export AUTH_REBAC_BACKEND=postgres

export AUTH_REBAC_SESSION_ENABLED=true

export AUTH_REBAC_SESSION_REDIS_URL=rediss://<redis-host>:6380/0

export AUTH_REBAC_SESSION_ALLOW_INSECURE=false  # set true only for localhost labs

# Tenant defaults & PAT tagging

export AUTH_REBAC_DEFAULT_TENANT_ID="__default__"  # single-tenant labs; omit for multi-tenant installs

export AUTH_PAT_TENANT_TAGGING_ENABLED=true        # ensures new PATs include tenant_id

# Optional: override only if you must disable automatic tuple writes (defaults to true when ReBAC is enabled)

# export AUTH_DATASET_OWNER_TUPLES_ENABLED=true

# ForwardAuth policy file (stateless requirement)

export AUTH_GATEWAY_POLICY_FILE=$KAMIWAZA_ROOT/config/auth_gateway_policy.yaml
```

Avoid stale overrides

If `env.sh` already contains older `AUTH_GATEWAY_*` exports, remove or comment them before adding the block above. Leaving legacy lines (for example, forcing `https://127.0.0.1` or generating a fresh client secret with `$(openssl rand -hex 32)`) overrides the helper output and will cause the Keycloak callback to fail. After running `run_oidc_uat.sh`, rely on the values written to `runtime/oidc-uat.env`.

The default `env.sh.example` ships with `AUTH_CALLBACK_URL=http://localhost:7777/api/auth/callback`. Update it to the HTTPS gateway URL (`https://<gateway-host>/api/auth/callback`) so the Keycloak redirect matches the public entrypoint.

Quick sanity check

1. Verify only the new `AUTH_GATEWAY_*` block exists in `env.sh` by running `grep AUTH_GATEWAY env.sh`. The helper-generated `runtime/oidc-uat.env` should report the same values.
2. Generate a single ForwardAuth secret and let the rotation helper copy it everywhere:

```bash
./scripts/rotate-auth-secrets.sh --secret AUTH_FORWARD_HEADER_SECRET --skip-compose
```

3. Rerun `./copy-compose.sh`, then bounce Traefik and the core stack:

```bash
sudo ./containers-down.sh traefik && sudo ./containers-up.sh traefik

./startup/kamiwazad.sh restart-core
```

4. Confirm every staged env file reports the same secret:

```bash
grep AUTH_FORWARD_HEADER_SECRET env.sh runtime/oidc-uat.env \

deployment/envs/${KAMIWAZA_ENV:-default}/kamiwaza-traefik/*/.env
```

If any of these values drift, Traefik forwards unsigned headers and the UI reports `forward_auth_signature_invalid` during model downloads.

Where to substitute values:

- `<keycloak-host>` – the external hostname for the Keycloak realm (for example `sso.example.com`).
- `<realm>` – the Keycloak realm that contains the Kamiwaza client (default `kamiwaza`).
- `<gateway-host>` – the public Traefik hostname for the Kamiwaza environment.
- `<client-secret>` – only required if the `kamiwaza-platform` client is configured as confidential with client-secret authentication.
- `<redis-host>` – the TLS endpoint of the Redis deployment used for ReBAC sessions (include a custom port if it is not `6380`).

After updating the file, reload it so the helper scripts and restart commands pick up the new values (especially `KAMIWAZA_KEYCLOAK_ADMIN_PASSWORD`).

```bash
source env.sh

source runtime/oidc-uat.env   # generated by scripts/run_oidc_uat.sh
```

`runtime/oidc-uat.env` is the source of truth for the Keycloak client metadata (issuer URL, JWKS path, client ID/secret). Never reintroduce older exports that point at different hosts or regenerate fresh secrets—those overrides are the most common root cause of `{"detail":"Authentication failed"}` when the gateway restarts.

Environment-specific Redis settings

Local labs usually run the bundled Redis instance without authentication. In that case, export:

```bash
export AUTH_REBAC_SESSION_ALLOW_INSECURE=true

export AUTH_REBAC_SESSION_REDIS_URL=redis://localhost:6379/0
```

As soon as you point the gateway at a secured or shared Redis deployment, switch to a credentialed URL—e.g. `rediss://user:pass@redis.example.com:6380/0`—and omit `AUTH_REBAC_SESSION_ALLOW_INSECURE`.

### Tenant fallback for labs

Stage‑1 bundles do not yet stamp `tenant_id` into the access token. Until the Keycloak protocol mapper is in place, allow the gateway to fall back to the `__default__` tenant by exporting:

```bash
export AUTH_REBAC_ALLOW_COMMUNITY_FALLBACK=true

export AUTH_REBAC_DEFAULT_TENANT_ID="__default__"
```

Only use these flags in local or UAT environments. As soon as tokens carry real tenant metadata, remove both exports so ReBAC enforces per-tenant policy.

### Logout redirect expectations

- Set `KAMIWAZA_UI_URL` to the full UI base (including any subpath) so logout returns users to the correct page. Example: `https://ui.example/app`.
- The gateway only trusts forwarded hosts that match the allowed hosts (UI URL or request host). Mismatched `X-Forwarded-Host` values are ignored.
- Localhost fallback (`https://localhost/login`) is used only when `KAMIWAZA_COMMUNITY` or `KAMIWAZA_LITE` is set. Without those flags and no valid base, logout returns HTTP 500 instead of redirecting to `/`.

### ReBAC manifest on RPM installs

- The RPM does not include `configs/rebac/tenants` or a default manifest. ReBAC still runs, but no tuples are preseeded. To bootstrap tuples, create `configs/rebac/tenants` under `/opt/kamiwaza/kamiwaza`, add a manifest (for example `__default__.yaml` from the repo/docs), and run your bootstrap/drift checks. Otherwise, you can run with an empty graph and add tuples later.

### Seed ReBAC tuples

Before operators can download or deploy a model they must be granted the appropriate tuples in the target tenant.

1. **Bootstrap the tenant manifest (recommended).**

```bash
python scripts/rebac_tenant.py bootstrap configs/rebac/tenants/__default__.yaml
```

Expected output:

```text
INFO:kamiwaza.services.auth.bootstrap:Applied tenant bootstrap

INFO:rebac_tenant:Tenant bootstrap applied successfully
```

Replace the manifest path with your tenant-specific file in production.
   - The manifests live in `configs/rebac/tenants/*.yaml`. Copy `__default__.yaml` to a tenant-specific file (for example `configs/rebac/tenants/acme-labs.yaml`) and edit the `relationships` section to declare owners, editors, and viewers for your resources.
   - Preview changes with `python scripts/rebac_tenant.py plan <manifest>` before applying them with `bootstrap`. The CLI is idempotent and can target a different tenant using `--tenant tenant-id`.
2. **Optionally add ad-hoc tuples** by editing the manifest and rerunning `plan`/`bootstrap`. Example entry:

```yaml
relationships:

- subject: user:analyst@example.com

relation: viewer

object: dataset:govdocs-ingest
```

Apply the change:

```bash
python scripts/rebac_tenant.py plan configs/rebac/tenants/acme-labs.yaml

python scripts/rebac_tenant.py bootstrap configs/rebac/tenants/acme-labs.yaml
```

Production environments should rely on the automatic owner helper (enabled by default when `AUTH_REBAC_ENABLED=true`) so catalog/model uploads seed owner/editor/viewer tuples without manual intervention. Use manifest edits for explicit overrides or cross-team sharing.

### quick smoke test

After the bootstrap, verify allow/deny flows before handing the stack to analysts:

1. Upload a dataset (or use an existing one) via the `/api/catalog/datasets/` endpoint.
2. Confirm list calls work:

```bash
curl -H "Authorization: Bearer $TOKEN" https://<gateway>/api/catalog/datasets/
```

Expected: HTTP `200` with the uploaded dataset URN in the JSON payload.
3. Confirm direct lookups succeed:

```bash
curl -H "Authorization: Bearer $TOKEN" \

"https://<gateway>/api/catalog/datasets/by-urn?urn=<dataset-urn>"
```

Expected: HTTP `200` with the dataset metadata. A `403 rebac_denied` indicates tuples are missing; re-run the bootstrap manifest.

Re-run the smoke test any time you rotate manifests or onboard a new tenant to ensure the guard coverage remains intact.

## Seed demo realm & client metadata

Run the helper to import the UAT realm, create the confidential client, and generate `runtime/oidc-uat.env` with matching values. If the packaged Keycloak is already running on the host, include `--no-start-keycloak` (to avoid port conflicts) and `--skip-install` (to bypass macOS bootstrap helpers).

```bash
./scripts/run_oidc_uat.sh --no-smoke --skip-install --no-start-keycloak \

--callback-url "https://<gateway-host>/api/auth/callback"
```

After generating the realm, reload your shell environment so the new client metadata is active for subsequent commands:

```bash
source env.sh

source runtime/oidc-uat.env
```

The helper writes `runtime/oidc-uat.env` with the gateway OIDC exports so follow-on scripts stay in sync. If you need to regenerate credentials (for example, after rotating secrets), rerun the helper with the same flags. Linux hosts can omit `--skip-install`; keep it on macOS to avoid rerunning the bootstrap installer. Keycloak administrator credentials continue to be managed by the install prelaunch step; they are not copied into that snapshot. Use the built-in utility (or read the secret file directly) to retrieve the value needed for the login test:

```bash
cat "${KAMIWAZA_ROOT:-$PWD}/runtime/secrets/keycloak-admin-password"
```

If the file is missing, ensure the packaged Keycloak stack has been started at least once (for example, `./containers-up.sh keycloak` or the equivalent systemd unit) so the prelaunch script can write the secret.

### Restart core services

Bounce the authentication plane so the new settings apply:

If you installed via the developer bundle, use the deployment helpers to bounce the containers so the new configuration is picked up. Run these from the repository root after `source env.sh`; set the swarm role explicitly (single-node labs export `KAMIWAZA_SWARM_HEAD=true`, multi-node environments can export `KAMIWAZA_HEAD_IP=<manager-ip>` instead). Use `sudo` if the helper needs to create `/etc/kamiwaza` swarm metadata or secrets:

```bash
export KAMIWAZA_SWARM_HEAD=${KAMIWAZA_SWARM_HEAD:-true}

sudo ./containers-down.sh keycloak

sudo ./containers-up.sh keycloak

sudo ./containers-down.sh traefik

sudo ./containers-up.sh traefik

./startup/kamiwazad.sh restart-core
```

If the helper reports the node is already part of a swarm, either reuse the existing manager by exporting `KAMIWAZA_HEAD_IP=<manager-ip>` before rerunning, or leave the old swarm (`docker swarm leave --force`) and re-run with `KAMIWAZA_SWARM_HEAD=true`.

If you are running the RPM or DEB services under systemd, restart the bundled service and confirm its status. Package installs also ship a convenience wrapper, `sudo kamiwaza restart-core`, which orchestrates the same steps.

```bash
sudo systemctl daemon-reload

sudo systemctl restart kamiwaza.service

sudo systemctl status kamiwaza.service --no-pager
```

If your site defines additional Kamiwaza units, list them first and restart only the ones that exist (for example, a custom Keycloak unit):

```bash
sudo systemctl list-units '*kamiwaza*' --no-pager

sudo systemctl restart kamiwaza-keycloak.service   # only if present
```

If you made changes to the deployment assets in `deployment/`, rerun `./copy-compose.sh` before the restart so the staged Compose bundles pick up the latest configuration.

### Using an external identity provider

Keycloak ships with the bundle to keep the walkthrough self-contained. To exercise a managed provider (Google Workspace, Okta, Azure AD, etc.), create an OIDC client there with the same redirect URI (`https://<gateway-host>/api/auth/callback`) and copy the issued client ID and secret into `env.sh` before sourcing it:

```bash
export AUTH_GATEWAY_KEYCLOAK_URL=https://accounts.google.com      # replace with provider base URL

export AUTH_GATEWAY_KEYCLOAK_REALM=<idp-tenant-or-realm>

export AUTH_GATEWAY_KEYCLOAK_CLIENT_ID=<issued-client-id>

export AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET=<issued-client-secret>

source env.sh
```

Skip `run_oidc_uat.sh` in that scenario; the managed IdP is now the source of truth. You still need the Redis exports from the earlier tip so the gateway can persist sessions.

## Verify login & header passthrough

1. Visit `https://<gateway-host>/api/auth/login` and sign in with the credentials seeded by the helper (defaults `admin` / `kamiwaza`; confirm the password by reading `runtime/secrets/keycloak-admin-password`).
2. Call the validation endpoint to ensure the session cookie works. Copy the `access_token` cookie from your browser (Developer Tools → Application → Cookies → `access_token`) and pass it in the request. Include `-k` when testing against a self-signed certificate.

```bash
curl -ki https://<gateway-host>/api/auth/validate \

-H "Cookie: access_token=<copied-session-cookie>"
```

Expect HTTP `200` with `X-User-*` headers.
3. If you receive a redirect loop, `401`, or `{"detail":"Authentication failed"}`, confirm `AUTH_GATEWAY_COOKIE_DOMAIN` matches the hostname **and** that every Traefik router attaches the `forwardauth@file` middleware:

```bash
rg -n "forwardauth" deployment/kamiwaza-traefik
```

The `forwardAuth` block must include `authRequestHeaders` (`Cookie`, `Authorization`) and the standard `authResponseHeaders` (`X-User-*` plus the signature fields). After editing, run `./copy-compose.sh`, `sudo ./containers-down.sh traefik`, `sudo ./containers-up.sh traefik`, and `./startup/kamiwazad.sh restart-core`.
4. If signatures still fail, resync the environment exports and restart the services:

```bash
source env.sh

source runtime/oidc-uat.env

./startup/kamiwazad.sh restart-core         # developer bundle

# or

sudo kamiwaza restart-core                  # RPM/DEB installs
```

This reloads the current client secret and clears any stale ForwardAuth state. If the admin password is unknown, rerun `scripts/run_oidc_uat.sh` with the same flags and re-check `runtime/secrets/keycloak-admin-password` before retrying the login.

## Next steps

- Walk through the [ReBAC validation checklist](https://docs.kamiwaza.ai/0.9.0/security/rebac-validation-checklist) to exercise tuple enforcement and decision logging.
- Review the [ReBAC overview](https://docs.kamiwaza.ai/0.9.0/security/rebac-overview) for architecture context.
- Coordinate with your security team to replace the demo realm with production IdP settings before go-live.

## Troubleshooting

- **`Authentication failed` during Keycloak login** – confirm the admin password by checking `KEYCLOAK_ADMIN_PASSWORD` in `runtime/oidc-uat.env`. If it does not match what you are entering, run `source env.sh` followed by `./scripts/run_oidc_uat.sh` with the same flags to regenerate the client and credentials.
- **`KAMIWAZA_KEYCLOAK_ADMIN_PASSWORD is missing` when restarting containers** – ensure you have run `source env.sh` in the current shell before invoking `docker compose` so the restart inherits the required variables. Re-run `copy-compose.sh` if you have not refreshed the deployment artifacts since editing `env.sh`.
- **Token exchange fails (`/api/auth/callback` returns 400/502)** – Keycloak is rejecting the client credentials. Verify `AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET` is not being overridden in `env.sh` (see the caution note above) and matches the secret shown in Keycloak for the `kamiwaza-platform` client. Re-run the helper (`./scripts/run_oidc_uat.sh …`), source `runtime/oidc-uat.env`, restart the services, and inspect `docker logs kamiwaza-keycloak-uat --tail 100` for `invalid_client` or redirect errors.
- **`forward_auth_signature_invalid` when downloading models** – Traefik is rejecting the gateway’s HMAC signature. Confirm `AUTH_FORWARD_HEADER_SECRET` matches across `env.sh`, `runtime/oidc-uat.env`, and `deployment/envs/${KAMIWAZA_ENV:-default}/kamiwaza-traefik/*/.env`. After updating the secret, rerun `./copy-compose.sh` and `./startup/kamiwazad.sh restart-core` so Traefik picks up the change.