Administrator Guide | Kamiwaza Docs

Documentation for Kamiwaza 0.8.0

This is documentation for Kamiwaza 0.8.0, which is no longer actively maintained. For the current GA release, see 1.0.1.

Version: 0.8.0

1. Authentication & Access Control

Kamiwaza provides enterprise-grade authentication built on Keycloak with OpenID Connect (OIDC) and JWT token validation.

1.1 Authentication Architecture

User → Keycloak (IdP) → JWT Token → Traefik → ForwardAuth → API Services

↓

[Validated] → Access Granted

↓

[Rejected] → 401/403 Error

Components:

Keycloak: Identity provider managing users, authentication, and token issuance
ForwardAuth Service: Validates JWT tokens and enforces access policies
Traefik: Reverse proxy routing requests through ForwardAuth middleware
RBAC Policy Engine: YAML-based endpoint access control

1.2 Authentication Modes

Kamiwaza supports two operational modes:

Mode	Use Case	Configuration
With Authentication	Production, staging, secure environments	`KAMIWAZA_USE_AUTH=true`
Bypass Mode	Local development, debugging	`KAMIWAZA_USE_AUTH=false`

To enable authentication:

# In env.sh or environment

export KAMIWAZA_USE_AUTH=true

bash startup/kamiwazad.sh restart

Expected output:

Stopping kamiwazad ...

Starting kamiwazad ...

kamiwazad status: active (running)

⚠️ Warning: Bypass mode (KAMIWAZA_USE_AUTH=false) disables all authentication. Use only in secure development environments.

1.3 Token-Based Authentication

Kamiwaza uses RS256 JWT tokens with asymmetric cryptographic signatures.

Token Lifecycle:

Acquisition: User authenticates with Keycloak via username/password or SSO
Validation: ForwardAuth validates token signature against JWKS endpoint
Authorization: User roles checked against RBAC policy
Expiration: Access tokens expire (default: 1 hour), require refresh
Revocation: Logout invalidates tokens

Token Delivery Methods:

HTTP Authorization: Bearer <token> header (recommended for APIs)
Secure HTTP-only cookie (automatic for browser sessions)

2. User Management

2.1 Manage Local Users in the Console

The Settings → Auth & Users screen is the fastest way to create local accounts.

Sign in to the Kamiwaza console with an administrator account.
Open Settings in the left nav and switch to the Auth & Users tab.
Click Add User.
Fill in the modal:
- Username – required login name.
- Full Name / Email – optional but recommended for auditing.
- Role – pick one of the built-in roles (viewer, user, admin). You can add multiple roles before saving.
- Password – enter the initial password and disable the “Must change password” toggle if this account needs to log in programmatically.
Click Save. The new user appears in the Local Users table.
Use the pencil icon to edit roles later, the key icon to reset passwords, and the trash can to remove the user.

Why disable “Must change password”? ReBAC smoke tests and SDK logins need to authenticate immediately. Leaving the toggle enabled causes Keycloak to demand a password reset on first login, resulting in an “Invalid credentials” error for CLIs and service accounts.

2.2 Configure External Identity Providers

If your organization uses Google Workspace or another OIDC provider:

In Settings → Auth & Users, switch to the Authentication Providers section.
Choose Google or Generic OIDC.
Supply the provider’s client ID, secret, and optional hosted domain.
Click Register. The new provider shows up under Configured Providers immediately—no restart required.

2.4 User Roles and Permissions

Kamiwaza defines three primary roles:

Role	Permissions	Typical Users
admin	Full access: read, write, delete, configure	System administrators, platform operators
user	Standard access: read, write (no delete/admin)	Data scientists, developers, analysts
viewer	Read-only access	Auditors, observers, stakeholders

Assigning Roles:

Navigate to Users → Select user
Go to Role Mappings tab
Under Realm Roles, select appropriate roles
Click Add selected
Changes take effect immediately (no logout required)

2.5 Password Policies

Configuring Password Requirements:

Navigate to Realm Settings → Security Defenses → Password Policy
Add policies:
- Minimum Length: 12 characters (recommended)
- Uppercase Characters: Require at least 1
- Lowercase Characters: Require at least 1
- Digits: Require at least 1
- Special Characters: Require at least 1
- Not Username: Prevent username as password
- Password History: Prevent last 3 passwords
- Expire Password: 90 days (recommended)

Password Reset Flow:

User clicks "Forgot Password" on login page
Keycloak sends password reset email
User follows link and sets new password
New password must meet policy requirements

2.6 Create a Local User in Lite Mode

Use this flow when KAMIWAZA_LITE=true and KAMIWAZA_USE_AUTH=false.

Set the admin password (required)

export KAMIWAZA_LITE=true

export KAMIWAZA_USE_AUTH=false

# Provide a password or allow generation (written under $KAMIWAZA_ROOT/runtime)

export KAMIWAZA_ADMIN_PASSWORD="kamiwaza"  # any >=12 chars for non-community builds

# export KAMIWAZA_ALLOW_GENERATED_ADMIN_PASSWORD=true  # optional fallback

Start services

bash launch.sh

Mint an admin bearer token (direct to core on port 7777)

ADMIN_TOKEN=$(curl -sk -X POST http://localhost:7777/api/auth/token \
  -H 'content-type: application/x-www-form-urlencoded' \
  -d 'grant_type=password' \
  -d 'client_id=kamiwaza-platform' \
  -d "username=admin" \
  -d "password=${KAMIWAZA_ADMIN_PASSWORD}" | jq -r '.access_token')

Create the user

curl -sk -X POST http://localhost:7777/api/auth/users/local \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "content-type: application/json" \
  -d '{"username":"demo","password":"demo12345678","email":"demo@example.com","roles":["user"]}' | jq

Verify

curl -sk http://localhost:7777/api/auth/users -H "Authorization: Bearer ${ADMIN_TOKEN}" | jq

Troubleshooting:

401/403 → missing or expired admin token.
Password changes not supported → account is external; use local users only in Lite.
KAMIWAZA_ADMIN_PASSWORD missing → set it or enable KAMIWAZA_ALLOW_GENERATED_ADMIN_PASSWORD=true and read runtime/generated-admin-password.txt.

3. Role-Based Access Control (RBAC)

3.1 RBAC Policy File

Access control is defined in YAML policy files that map endpoints to required roles.

Default Location:

Host installs: $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml
Docker installs: Mounted at /app/config/auth_gateway_policy.yaml

ForwardAuth is stateless—set AUTH_GATEWAY_POLICY_FILE=$KAMIWAZA_ROOT/config/auth_gateway_policy.yaml (or the mounted path) so every restart reloads the same policy file.

Policy File Structure (default config/auth_gateway_policy.yaml):

version: 1

env: dev

default_deny: true

roles:

- id: admin

description: "Full system access"

- id: user

description: "Standard user access"

- id: viewer

description: "Read-only access"

- id: guest

description: "Minimal guest access"

endpoints:

# Health checks

- path: "/health"

methods: ["GET"]

roles: ["*"]

- path: "/api/health"

methods: ["GET"]

roles: ["*"]

# Auth endpoints (login/logout)

- path: "/auth/login"

methods: ["POST"]

roles: ["*"]

- path: "/auth/logout"

methods: ["POST"]

roles: ["*"]

# Who am I

- path: "/api/whoami"

methods: ["GET"]

roles: ["admin", "user", "viewer", "guest"]

# Models

- path: "/api/models*"

methods: ["GET"]

roles: ["admin", "user", "viewer"]

- path: "/api/models*"

methods: ["POST", "PUT", "DELETE"]

roles: ["admin", "user"]

# Serving deployments

- path: "/api/serving/deployments*"

methods: ["GET"]

roles: ["admin", "user", "viewer"]

- path: "/api/serving/deployments*"

methods: ["POST", "PUT", "DELETE"]

roles: ["admin", "user"]

# Admin-only APIs

- path: "/api/cluster*"

methods: ["*"]

roles: ["admin"]

- path: "/api/activity*"

methods: ["*"]

roles: ["admin"]

# Garden apps + tools

- path: "/api/apps*"

methods: ["GET"]

roles: ["admin", "user", "viewer"]

- path: "/api/apps*"

methods: ["POST", "PUT", "DELETE"]

roles: ["admin", "user"]

- path: "/api/tools*"

methods: ["GET"]

roles: ["admin", "user", "viewer"]

- path: "/api/tools*"

methods: ["POST", "PUT", "DELETE"]

roles: ["admin", "user"]

# Data Discovery Engine

- path: "/api/dde/status"

methods: ["GET"]

roles: ["viewer", "admin"]

- path: "/api/dde/search"

methods: ["POST"]

roles: ["viewer", "admin"]

- path: "/api/dde/reindex"

methods: ["POST"]

roles: ["admin"]

# Static assets

- path: "/static/*"

methods: ["GET"]

roles: ["admin", "user", "viewer", "guest"]

- path: "/assets/*"

methods: ["GET"]

roles: ["admin", "user", "viewer", "guest"]

- path: "/favicon.ico"

methods: ["GET"]

roles: ["admin", "user", "viewer", "guest"]

- path: "/manifest.json"

methods: ["GET"]

roles: ["admin", "user", "viewer", "guest"]

3.3 Path Matching Rules

Wildcard Patterns:

* matches zero or more characters within a path segment
** matches across multiple path segments
Patterns are case-sensitive

3.4 Adding Custom Endpoints

Example: Protecting a new analytics endpoint

endpoints:

# Add new analytics endpoint

- path: "/api/analytics/reports*"

methods: ["GET"]

roles: ["user", "admin"]

- path: "/api/analytics/reports*"

methods: ["POST", "DELETE"]

roles: ["admin"]

4. Identity Provider Integration

4.1 Keycloak Configuration

Realm:kamiwaza Client ID:kamiwaza-platform

Client Configuration Settings:

Setting	Value	Purpose
Access Type	Public (SPA) or Confidential (backend)	Authentication flow type
Valid Redirect URIs	`https://your-domain.com/*`	Allowed OAuth callback URLs
Web Origins	`https://your-domain.com`	CORS configuration
Direct Access Grants	Enabled (dev), Disabled (prod)	Password grant for testing

4.2 OAuth 2.0 / OpenID Connect Integration

Kamiwaza supports standard OIDC authentication flows.

Environment Configuration:

# Keycloak OIDC Settings

AUTH_GATEWAY_KEYCLOAK_URL=https://auth.yourdomain.com

AUTH_GATEWAY_KEYCLOAK_REALM=kamiwaza

AUTH_GATEWAY_KEYCLOAK_CLIENT_ID=kamiwaza-platform

# JWT Validation

AUTH_GATEWAY_JWT_ISSUER=https://auth.yourdomain.com/realms/kamiwaza

AUTH_GATEWAY_JWT_AUDIENCE=kamiwaza-platform

AUTH_GATEWAY_JWKS_URL=https://auth.yourdomain.com/realms/kamiwaza/protocol/openid-connect/certs

4.3 SAML Integration

Configure SAML Identity Provider in Keycloak:

Navigate to Identity Providers in Keycloak admin console
Select SAML v2.0
Configure SAML settings:
- Single Sign-On Service URL: Your IdP's SSO endpoint
- Single Logout Service URL: Your IdP's logout endpoint
- NameID Policy Format: urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
- Principal Type: Subject NameID
Upload IdP metadata XML or configure manually
Map SAML attributes to Keycloak user attributes
Enable identity provider in login flow

4.4 LDAP / Active Directory Integration

Configure LDAP Federation:

Navigate to User Federation → Add provider → ldap
Configure connection settings:
- Connection URL: ldap://ldap.company.com:389 or ldaps:// for SSL
- Bind DN: cn=admin,dc=company,dc=com
- Bind Credential: LDAP admin password
Configure LDAP search settings:
- Users DN: ou=users,dc=company,dc=com
- User Object Classes: inetOrgPerson, organizationalPerson
- Username LDAP attribute: uid or sAMAccountName (AD)
- RDN LDAP attribute: uid or cn
- UUID LDAP attribute: entryUUID or objectGUID (AD)
Save and test connection
Synchronize users: Synchronize all users button

4.5 Single Sign-On (SSO) Setup

Google SSO Integration:

Create OAuth 2.0 credentials in Google Cloud Console
Configure authorized redirect URI:

https://auth.yourdomain.com/realms/kamiwaza/broker/google/endpoint

In Keycloak, navigate to Identity Providers → Google
Enter Client ID and Client Secret from Google Console
Save and enable

Testing SSO:

Navigate to Kamiwaza login page
Click SSO provider button (Google, Azure, etc.)
Authenticate with external identity provider
First-time users automatically create Keycloak account
Subsequent logins use existing account

5. Security Configuration

5.1 JWT Token Configuration

Token Security Settings:

# JWT Validation (in env.sh)

AUTH_GATEWAY_JWT_AUDIENCE=kamiwaza-platform  # Required audience claim

AUTH_GATEWAY_JWT_ISSUER=https://auth.yourdomain.com/realms/kamiwaza

AUTH_GATEWAY_JWKS_URL=https://auth.yourdomain.com/realms/kamiwaza/protocol/openid-connect/certs

# Security Hardening

AUTH_REQUIRE_SUB=true  # Require 'sub' claim (user ID) in tokens

AUTH_EXPOSE_TOKEN_HEADER=false  # Don't expose tokens in response headers (production)

AUTH_ALLOW_UNSIGNED_STATE=false  # Require signed OIDC state parameter (production)

5.2 Session Management

Access Token Expiration:

Configure in Keycloak: Realm Settings → Tokens

Access Token Lifespan: 1 hour (default), 5-15 minutes (high security)
Refresh Token Lifespan: 30 days (default)
SSO Session Idle: 30 minutes
SSO Session Max: 10 hours

Best Practices:

Short-lived access tokens (5-15 minutes) for high-security environments
Longer refresh tokens (days) for user convenience
Implement token refresh in client applications
Use secure, HTTP-only cookies for browser sessions

5.4 Rate Limiting (Optional - Requires Redis)

Rate limiting requires Redis configuration:

# Redis connection for rate limiting

REDIS_HOST=localhost

REDIS_PORT=6379

REDIS_DB=0

Rate Limit Configuration:

# In auth_gateway_policy.yaml

rate_limits:

- path: "/api/models*"

requests_per_minute: 100

per_user: true

- path: "/api/auth/token"

requests_per_minute: 10

per_ip: true

6. Monitoring & Troubleshooting

6.1 Health Checks

Auth Service Health Endpoint:

curl http://localhost:7777/health

Response:

{

"status": "healthy",

"version": "1.0.0",

"uptime": 3600.5,

"KAMIWAZA_USE_AUTH": true,

"jwks_cache_status": "healthy"
}

Keycloak Health Check:

curl http://localhost:8080/health/ready

Response:

{"status":"UP"}

6.3 Common Issues and Solutions

Issue: 401 Unauthorized on All Requests

Symptoms: All API requests return 401 even with valid tokens

Troubleshooting:

Check if auth is enabled:

echo $KAMIWAZA_USE_AUTH  # Should be 'true'

Verify Keycloak is running:

docker ps | grep keycloak

curl http://localhost:8080/health/ready

Issue: 403 Forbidden (Valid Token)

Symptoms: Token is valid but access denied

Troubleshooting:

Check user roles in token:

echo $TOKEN | cut -d. -f2 | base64 -d | jq .realm_access.roles

Verify RBAC policy allows access:

cat $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

Check policy file syntax:

# Invalid YAML prevents policy reload

yamllint $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

6.4 Diagnostic Commands

Test Token Generation:

# Get token from Keycloak

TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=password" \
  -d "client_id=kamiwaza-platform" \
  -d "username=testuser" \
  -d "password=testpass" | jq -r .access_token)

# Decode token to inspect claims

echo $TOKEN | cut -d. -f2 | base64 -d | jq .

Verify Keycloak login flows

Use these checks after configuring SAML/OIDC to confirm the gateway and Keycloak agree on redirect URIs and credentials.

OIDC loop

curl -I https://<gateway-host>/api/auth/login

Appendix A: Environment Variable Reference

Variable	Description	Default	Required
`KAMIWAZA_USE_AUTH`	Enable/disable authentication	`true`	No
`AUTH_GATEWAY_JWT_ISSUER`	Expected JWT issuer URL	-	Yes
`AUTH_GATEWAY_JWT_AUDIENCE`	Expected JWT audience claim	-	Recommended
`AUTH_GATEWAY_JWKS_URL`	JWKS endpoint for key fetching	-	Yes
`AUTH_GATEWAY_POLICY_FILE`	Path to RBAC policy file	`$KAMIWAZA_ROOT/config/auth_gateway_policy.yaml`	No

Appendix B: RBAC Policy Examples

Example 1: Tiered Access by Service

version: 1

env: production

default_deny: true

roles:

- id: admin

description: "System administrators"

- id: data_scientist

description: "Data scientists and ML engineers"

- id: analyst

description: "Business analysts and viewers"

endpoints:

# Model Management - Scientists can create/edit, analysts read-only

- path: "/api/models*"

methods: ["GET"]

roles: ["admin", "data_scientist", "analyst"]

- path: "/api/models*"

methods: ["POST", "PUT", "DELETE"]

roles: ["admin", "data_scientist"]

# Public endpoints

- path: "/health"

methods: ["GET"]

roles: ["*"]

Example 2: Read-Write Separation

version: 1

env: production

default_deny: true

roles:

- id: admin

- id: editor

- id: reader

endpoints:

# Read endpoints - All authenticated users

- path: "/api/models"

methods: ["GET"]

roles: ["admin", "editor", "reader"]

# Write endpoints - Editors and admins only

- path: "/api/models"

methods: ["POST", "PUT"]

roles: ["admin", "editor"]

# Delete endpoints - Admins only

- path: "/api/models*"

methods: ["DELETE"]

roles: ["admin"]