Administrator Guide | Kamiwaza Docs

Documentation for Kamiwaza 0.6.0

This is documentation for Kamiwaza 0.6.0, which is no longer actively maintained. For the current GA release, see 1.0.1.

Version: 0.6.0

1. Authentication & Access Control

Kamiwaza provides enterprise-grade authentication built on Keycloak with OpenID Connect (OIDC) and JWT token validation.

1.1 Authentication Architecture

User → Keycloak (IdP) → JWT Token → Traefik → ForwardAuth → API Services

↓

[Validated] → Access Granted

↓

[Rejected] → 401/403 Error

Components:

Keycloak: Identity provider managing users, authentication, and token issuance
ForwardAuth Service: Validates JWT tokens and enforces access policies
Traefik: Reverse proxy routing requests through ForwardAuth middleware
RBAC Policy Engine: YAML-based endpoint access control

1.2 Authentication Modes

Kamiwaza supports two operational modes:

Mode	Use Case	Configuration
With Authentication	Production, staging, secure environments	`KAMIWAZA_USE_AUTH=true`
Bypass Mode	Local development, debugging	`KAMIWAZA_USE_AUTH=false`

To enable authentication:

# In env.sh or environment

export KAMIWAZA_USE_AUTH=true

bash startup/kamiwazad.sh restart

⚠️ Warning: Bypass mode (KAMIWAZA_USE_AUTH=false) disables all authentication. Use only in secure development environments.

1.3 Token-Based Authentication

Kamiwaza uses RS256 JWT tokens with asymmetric cryptographic signatures.

Token Lifecycle:

Acquisition: User authenticates with Keycloak via username/password or SSO
Validation: ForwardAuth validates token signature against JWKS endpoint
Authorization: User roles checked against RBAC policy
Expiration: Access tokens expire (default: 1 hour), require refresh
Revocation: Logout invalidates tokens

Token Delivery Methods:

HTTP Authorization: Bearer <token> header (recommended for APIs)
Secure HTTP-only cookie (automatic for browser sessions)

2. User Management

2.1 Accessing Keycloak Admin Console

Default Credentials (change immediately in production):

URL: http://localhost:8080 (or your configured Keycloak URL)
Username: admin
Password: Set via KEYCLOAK_ADMIN_PASSWORD environment variable

Production Setup:

# Set secure admin password in env.sh

export KEYCLOAK_ADMIN_PASSWORD="<strong-random-password>"

2.2 Creating User Accounts

Via Keycloak Admin Console:

Navigate to Users in left sidebar
Click Add User
Fill in required fields:
- Username (required)
- Email (required for password reset)
- First Name / Last Name (optional)
Toggle Email Verified to ON
Click Save
Go to Credentials tab
Set temporary or permanent password
Assign roles (see Role Management below)

Pre-configured Test Users:

Username	Password	Roles	Use Case
`testuser`	`testpass`	viewer	Read-only testing
`testadmin`	`testpass`	admin	Administrative testing

⚠️ Important: Remove or secure test users before production deployment.

2.3 User Roles and Permissions

Kamiwaza defines three primary roles:

Role	Permissions	Typical Users
admin	Full access: read, write, delete, configure	System administrators, platform operators
user	Standard access: read, write (no delete/admin)	Data scientists, developers, analysts
viewer	Read-only access	Auditors, observers, stakeholders

Assigning Roles:

Navigate to Users → Select user
Go to Role Mappings tab
Under Realm Roles, select appropriate roles
Click Add selected
Changes take effect immediately (no logout required)

2.4 Password Policies

Configuring Password Requirements:

Navigate to Realm Settings → Security Defenses → Password Policy
Add policies:
- Minimum Length: 12 characters (recommended)
- Uppercase Characters: Require at least 1
- Lowercase Characters: Require at least 1
- Digits: Require at least 1
- Special Characters: Require at least 1
- Not Username: Prevent username as password
- Password History: Prevent last 3 passwords
- Expire Password: 90 days (recommended)

Password Reset Flow:

User clicks "Forgot Password" on login page
Keycloak sends password reset email
User follows link and sets new password
New password must meet policy requirements

⚠️ Important: Configure SMTP settings in Keycloak for email-based password reset to function.

3. Role-Based Access Control (RBAC)

3.1 RBAC Policy File

Access control is defined in YAML policy files that map endpoints to required roles.

Default Location:

Host installs: $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml
Docker installs: Mounted at /app/config/auth_gateway_policy.yaml

Policy File Structure:

version: 1

env: production

default_deny: true  # Block all endpoints unless explicitly allowed

roles:

- id: admin

description: "Full system access"

- id: user

description: "Standard user access"

- id: viewer

description: "Read-only access"

endpoints:

# Model Management

- path: "/api/models*"

methods: ["GET"]

roles: ["viewer", "user", "admin"]

- path: "/api/models*"

methods: ["POST", "PUT", "DELETE"]

roles: ["user", "admin"]

# Cluster Management (Admin-only)

- path: "/api/cluster*"

methods: ["*"]

roles: ["admin"]

# Vector Database (User and Admin)

- path: "/api/vectordb*"

methods: ["GET"]

roles: ["viewer", "user", "admin"]

- path: "/api/vectordb*"

methods: ["POST", "PUT", "DELETE"]

roles: ["user", "admin"]

# Public endpoints (no auth required)

- path: "/health"

methods: ["GET"]

roles: ["*"]  # Public

- path: "/docs"

methods: ["GET"]

roles: ["*"]  # Public API documentation

3.2 Path Matching Rules

Wildcard Patterns:

* matches zero or more characters within a path segment
** matches across multiple path segments
Patterns are case-sensitive

Examples:

/api/models* matches /api/models, /api/models/123, /api/models/search
/api/*/health matches /api/models/health, /api/cluster/health
/api/** matches all paths under /api/

3.3 Hot Reload (No Restart Required)

The RBAC policy file is automatically reloaded when modified:

Edit auth_gateway_policy.yaml
Save the file
Changes take effect within seconds
Monitor logs for reload confirmation:

INFO: Policy reloaded successfully from /app/config/auth_gateway_policy.yaml

⚠️ Important: Invalid YAML syntax will prevent reload and retain the previous valid configuration.

3.4 Adding Custom Endpoints

Example: Protecting a new analytics endpoint

endpoints:

# Add new analytics endpoint

- path: "/api/analytics/reports*"

methods: ["GET"]

roles: ["user", "admin"]

- path: "/api/analytics/reports*"

methods: ["POST", "DELETE"]

roles: ["admin"]

Testing Access Control:

# Get token for viewer role (should be denied POST)

VIEWER_TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \

-d "grant_type=password" \

-d "client_id=kamiwaza-platform" \

-d "username=testuser" \

-d "password=testpass" | jq -r .access_token)

# Test (expect 403 Forbidden)

curl -H "Authorization: Bearer $VIEWER_TOKEN" \

-X POST http://localhost:7777/api/analytics/reports

# Get token for admin role (should succeed)

ADMIN_TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \

-d "grant_type=password" \

-d "client_id=kamiwaza-platform" \

-d "username=testadmin" \

-d "password=testpass" | jq -r .access_token)

# Test (expect 200 OK)

curl -H "Authorization: Bearer $ADMIN_TOKEN" \

-X POST http://localhost:7777/api/analytics/reports

4. Identity Provider Integration

4.1 Keycloak Configuration

Realm: kamiwaza Client ID: kamiwaza-platform

Client Configuration Settings:

Setting	Value	Purpose
Access Type	Public (SPA) or Confidential (backend)	Authentication flow type
Valid Redirect URIs	`https://your-domain.com/*`	Allowed OAuth callback URLs
Web Origins	`https://your-domain.com`	CORS configuration
Direct Access Grants	Enabled (dev), Disabled (prod)	Password grant for testing

4.2 OAuth 2.0 / OpenID Connect Integration

Kamiwaza supports standard OIDC authentication flows.

Environment Configuration:

# Keycloak OIDC Settings

AUTH_GATEWAY_KEYCLOAK_URL=https://auth.yourdomain.com

AUTH_GATEWAY_KEYCLOAK_REALM=kamiwaza

AUTH_GATEWAY_KEYCLOAK_CLIENT_ID=kamiwaza-platform

# JWT Validation

AUTH_GATEWAY_JWT_ISSUER=https://auth.yourdomain.com/realms/kamiwaza

AUTH_GATEWAY_JWT_AUDIENCE=kamiwaza-platform

AUTH_GATEWAY_JWKS_URL=https://auth.yourdomain.com/realms/kamiwaza/protocol/openid-connect/certs

OIDC Discovery Endpoint:

https://auth.yourdomain.com/realms/kamiwaza/.well-known/openid-configuration

4.3 SAML Integration

Configure SAML Identity Provider in Keycloak:

Navigate to Identity Providers in Keycloak admin console
Select SAML v2.0
Configure SAML settings:
- Single Sign-On Service URL: Your IdP's SSO endpoint
- Single Logout Service URL: Your IdP's logout endpoint
- NameID Policy Format: urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
- Principal Type: Subject NameID
Upload IdP metadata XML or configure manually
Map SAML attributes to Keycloak user attributes
Enable identity provider in login flow

Attribute Mapping Example:

SAML Attribute        → Keycloak Attribute

-----------------       -------------------

email                 → email

firstName             → firstName

lastName              → lastName

memberOf              → roles

4.4 LDAP / Active Directory Integration

Configure LDAP Federation:

Navigate to User Federation → Add provider → ldap
Configure connection settings:
- Connection URL: ldap://ldap.company.com:389 or ldaps:// for SSL
- Bind DN: cn=admin,dc=company,dc=com
- Bind Credential: LDAP admin password
Configure LDAP search settings:
- Users DN: ou=users,dc=company,dc=com
- User Object Classes: inetOrgPerson, organizationalPerson
- Username LDAP attribute: uid or sAMAccountName (AD)
- RDN LDAP attribute: uid or cn
- UUID LDAP attribute: entryUUID or objectGUID (AD)
Save and test connection
Synchronize users: Synchronize all users button

Active Directory Specific Settings:

Vendor: Active Directory
Username LDAP attribute: sAMAccountName
RDN LDAP attribute: cn
UUID LDAP attribute: objectGUID
User Object Classes: person, organizationalPerson, user

Role Mapping from LDAP Groups:

Go to Mappers tab in LDAP federation
Create new mapper: group-ldap-mapper
- Mapper Type: group-ldap-mapper
- LDAP Groups DN: ou=groups,dc=company,dc=com
- Group Name LDAP Attribute: cn
- Group Object Classes: groupOfNames
- Membership LDAP Attribute: member
- Mode: READ_ONLY or LDAP_ONLY
Map LDAP groups to Keycloak roles in Role Mappings

4.5 Single Sign-On (SSO) Setup

Google SSO Integration:

Create OAuth 2.0 credentials in Google Cloud Console
Configure authorized redirect URI:

https://auth.yourdomain.com/realms/kamiwaza/broker/google/endpoint

In Keycloak, navigate to Identity Providers → Google
Enter Client ID and Client Secret from Google Console
Save and enable

Environment Configuration:

# Google SSO

GOOGLE_CLIENT_ID=your-google-client-id

GOOGLE_CLIENT_SECRET=your-google-client-secret

Microsoft Azure AD / Office 365:

Register application in Azure Portal
Configure redirect URI: https://auth.yourdomain.com/realms/kamiwaza/broker/oidc/endpoint
In Keycloak, add OpenID Connect v1.0 provider
Configure with Azure AD settings:
- Authorization URL: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize
- Token URL: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
- Client ID: Azure application ID
- Client Secret: Azure client secret

Testing SSO:

Navigate to Kamiwaza login page
Click SSO provider button (Google, Azure, etc.)
Authenticate with external identity provider
First-time users automatically create Keycloak account
Subsequent logins use existing account

5. Security Configuration

5.1 JWT Token Configuration

Token Security Settings:

# JWT Validation (in env.sh)

aUTH_GATEWAY_JWT_AUDIENCE=kamiwaza-platform  # Required audience claim

aUTH_GATEWAY_JWT_ISSUER=https://auth.yourdomain.com/realms/kamiwaza

aUTH_GATEWAY_JWKS_URL=https://auth.yourdomain.com/realms/kamiwaza/protocol/openid-connect/certs

# Security Hardening
aUTH_REQUIRE_SUB=true  # Require 'sub' claim (user ID) in tokens

aUTH_EXPOSE_TOKEN_HEADER=false  # Don't expose tokens in response headers (production)

aUTH_ALLOW_UNSIGNED_STATE=false  # Require signed OIDC state parameter (production)

Token Algorithms:

Supported: RS256 (RSA with SHA-256) - asymmetric cryptography
Not Supported: HS256, ES256, or other algorithms

5.2 Session Management

Access Token Expiration:

Configure in Keycloak: Realm Settings → Tokens

Access Token Lifespan: 1 hour (default), 5-15 minutes (high security)
Refresh Token Lifespan: 30 days (default)
SSO Session Idle: 30 minutes
SSO Session Max: 10 hours

Session Timeout Configuration:

# In env.sh

aUTH_GATEWAY_TOKEN_LEEWAY=30  # Clock skew tolerance (seconds)

aUTH_GATEWAY_JWKS_CACHE_TTL=300  # JWKS cache duration (5 minutes)

Best Practices:

Short-lived access tokens (5-15 minutes) for high-security environments
Longer refresh tokens (days) for user convenience
Implement token refresh in client applications
Use secure, HTTP-only cookies for browser sessions

5.3 HTTPS Enforcement

Production HTTPS Requirements:

Kamiwaza enforces HTTPS in production and CI environments when CI=true or KAMIWAZA_ENV=production.

TLS Configuration:

Obtain SSL/TLS certificates (Let's Encrypt, commercial CA, etc.)
Configure Traefik with TLS:

# traefik-dynamic.yml
tls:

certificates:

- certFile: /certs/your-domain.crt

keyFile: /certs/your-domain.key

options:

default:

minVersion: VersionTLS12

cipherSuites:

- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Update environment:

AUTH_GATEWAY_KEYCLOAK_URL=https://auth.yourdomain.com

KAMIWAZA_HTTPS=true

5.4 Rate Limiting (Optional - Requires Redis)

Rate limiting requires Redis configuration:

# Redis connection for rate limiting

REDIS_HOST=localhost

REDIS_PORT=6379

REDIS_DB=0

Rate Limit Configuration:

# In auth_gateway_policy.yaml

rate_limits:

- path: "/api/models*"

requests_per_minute: 100

per_user: true

- path: "/api/auth/token"

requests_per_minute: 10

per_ip: true

6. Monitoring & Troubleshooting

6.1 Health Checks

Auth Service Health Endpoint:

curl http://localhost:7777/health

Response:

{
  "status": "healthy",
  "version": "1.0.0",
  "uptime": 3600.5,
  "KAMIWAZA_USE_AUTH": true,
  "jwks_cache_status": "healthy"
}

Keycloak Health Check:

curl http://localhost:8080/health/ready

6.2 Log Monitoring

Auth Service Logs:

# Docker deployments
docker logs kamiwaza-api -f | grep AUTH

# Host deployments
tail -f $KAMIWAZA_LOG_DIR/kamiwaza.log | grep AUTH

Important Log Events:

AUTH_FAILED - Authentication failure with reason
ACCESS_DENIED - Authorization denial with path/method/roles
JWKS_REFRESHED - JWKS key cache refresh
POLICY_RELOADED - RBAC policy file reload
TOKEN_VALIDATED - Successful token validation

6.3 Common Issues and Solutions

Issue: 401 Unauthorized on All Requests

Symptoms: All API requests return 401 even with valid tokens

Troubleshooting:

Check if auth is enabled:

echo $KAMIWAZA_USE_AUTH  # Should be 'true'

Verify Keycloak is running:

docker ps | grep keycloak

curl http://localhost:8080/health/ready

Check JWT issuer matches:

# Decode your token
echo $TOKEN | cut -d. -f2 | base64 -d | jq .iss

# Compare with configuration
echo $AUTH_GATEWAY_JWT_ISSUER

Verify JWKS endpoint is accessible:

curl $AUTH_GATEWAY_JWKS_URL

Solution:

Ensure AUTH_GATEWAY_JWT_ISSUER matches token issuer exactly
Verify Keycloak realm name is correct
Check network connectivity to Keycloak

Issue: 403 Forbidden (Valid Token)

Symptoms: Token is valid but access denied

Troubleshooting:

Check user roles in token:

echo $TOKEN | cut -d. -f2 | base64 -d | jq .realm_access.roles

Verify RBAC policy allows access:

cat $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

Check policy file syntax:

# Invalid YAML prevents policy reload

yamllint $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

Solution:

Add required roles to user in Keycloak
Update RBAC policy to allow endpoint/method/role combination
Fix YAML syntax errors and reload policy

Issue: Token Expired Too Quickly

Symptoms: Tokens expire after minutes instead of expected duration

Troubleshooting:

Check token lifespan in Keycloak:
- Navigate to Realm Settings → Tokens
- Verify Access Token Lifespan setting
Check token claims:

echo $TOKEN | cut -d. -f2 | base64 -d | jq '.exp - .iat'

# Result is token lifetime in seconds

Solution:

Increase Access Token Lifespan in Keycloak (for development)
Implement token refresh in client applications
Use refresh tokens for long-lived sessions

Issue: Google/SSO Login Not Working

Symptoms: SSO redirect fails or returns error

Troubleshooting:

Check redirect URI configuration:
- Verify redirect URI in Google/Azure console matches Keycloak exactly
- Format: https://auth.yourdomain.com/realms/kamiwaza/broker/{provider}/endpoint
Verify client secret is set:

echo $GOOGLE_CLIENT_SECRET  # Should not be empty

Check Keycloak identity provider logs:

docker logs kamiwaza-keycloak -f | grep -i broker

Solution:

Update authorized redirect URIs in OAuth provider console
Ensure client secret is configured in Keycloak
Enable identity provider in Keycloak authentication flow

6.4 Diagnostic Commands

Test Token Generation:

# Get token from Keycloak

TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \

-H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=password" \
  -d "client_id=kamiwaza-platform" \
  -d "username=testuser" \
  -d "password=testpass" | jq -r .access_token)

# Decode token to inspect claims
echo $TOKEN | cut -d. -f2 | base64 -d | jq .

Test Token Validation:

# Test ForwardAuth validation endpoint directly

curl -v -H "Authorization: Bearer $TOKEN" \
  -H "X-Forwarded-Uri: /api/models" \
  -H "X-Forwarded-Method: GET" \
  http://localhost:7777/auth/validate

Verify JWKS Endpoint:

# Fetch public keys for signature validation

curl http://localhost:8080/realms/kamiwaza/protocol/openid-connect/certs | jq .

Check RBAC Policy:

# View current policy

cat $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

# Watch for policy reload events
tail -f $KAMIWAZA_LOG_DIR/kamiwaza.log | grep POLICY_RELOADED

Appendix A: Environment Variable Reference

Core Authentication

Variable	Description	Default	Required
`KAMIWAZA_USE_AUTH`	Enable/disable authentication	`true`	No
`AUTH_GATEWAY_JWT_ISSUER`	Expected JWT issuer URL	-	Yes
`AUTH_GATEWAY_JWT_AUDIENCE`	Expected JWT audience claim	-	Recommended
`AUTH_GATEWAY_JWKS_URL`	JWKS endpoint for key fetching	-	Yes
`AUTH_GATEWAY_POLICY_FILE`	Path to RBAC policy file	`$KAMIWAZA_ROOT/config/auth_gateway_policy.yaml`	No

Keycloak Configuration

Variable	Description	Default	Required
`AUTH_GATEWAY_KEYCLOAK_URL`	Keycloak base URL	`http://localhost:8080`	Yes
`AUTH_GATEWAY_KEYCLOAK_REALM`	Keycloak realm name	`kamiwaza`	Yes
`AUTH_GATEWAY_KEYCLOAK_CLIENT_ID`	OAuth client ID	`kamiwaza-platform`	Yes
`KEYCLOAK_ADMIN_PASSWORD`	Keycloak admin password	`admin`	Yes

Security Hardening

Variable	Description	Default	Required
`AUTH_REQUIRE_SUB`	Require 'sub' claim in tokens	`false`	No
`AUTH_EXPOSE_TOKEN_HEADER`	Expose token in response headers	`true`	No
`AUTH_ALLOW_UNSIGNED_STATE`	Allow unsigned OIDC state	`true` (dev only)	No
`AUTH_GATEWAY_TOKEN_LEEWAY`	Clock skew tolerance (seconds)	`30`	No
`AUTH_GATEWAY_JWKS_CACHE_TTL`	JWKS cache duration (seconds)	`300`	No

External Identity Providers

Variable	Description	Default	Required
`GOOGLE_CLIENT_ID`	Google OAuth client ID	-	For Google SSO
`GOOGLE_CLIENT_SECRET`	Google OAuth client secret	-	For Google SSO

Appendix B: RBAC Policy Examples

Example 1: Tiered Access by Service

version: 1

env: production

default_deny: true

roles:

- id: admin

description: "System administrators"

- id: data_scientist

description: "Data scientists and ML engineers"

- id: analyst

description: "Business analysts and viewers"

endpoints:

# Model Management - Scientists can create/edit, analysts read-only

- path: "/api/models*"

methods: ["GET"]

roles: ["admin", "data_scientist", "analyst"]

- path: "/api/models*"

methods: ["POST", "PUT", "DELETE"]

roles: ["admin", "data_scientist"]

# Model Serving - Scientists can deploy, analysts can query

- path: "/api/serving/deployments*"

methods: ["GET"]

roles: ["admin", "data_scientist", "analyst"]

- path: "/api/serving/deploy"

methods: ["POST"]

roles: ["admin", "data_scientist"]

- path: "/api/serving/generate"

methods: ["POST"]

roles: ["admin", "data_scientist", "analyst"]

# Cluster Management - Admin-only

- path: "/api/cluster*"

methods: ["*"]

roles: ["admin"]

# Public endpoints

- path: "/health"

methods: ["GET"]

roles: ["*"]

Example 2: Read-Write Separation

version: 1

env: production

default_deny: true

roles:

- id: admin

- id: editor

- id: reader

endpoints:

# Read endpoints - All authenticated users

- path: "/api/models"

methods: ["GET"]

roles: ["admin", "editor", "reader"]

- path: "/api/vectordb/collections"

methods: ["GET"]

roles: ["admin", "editor", "reader"]

# Write endpoints - Editors and admins only

- path: "/api/models"

methods: ["POST", "PUT"]

roles: ["admin", "editor"]

- path: "/api/vectordb/collections"

methods: ["POST", "PUT"]

roles: ["admin", "editor"]

# Delete endpoints - Admins only

- path: "/api/models*"

methods: ["DELETE"]

roles: ["admin"]

- path: "/api/vectordb/collections*"

methods: ["DELETE"]

roles: ["admin"]