Documentation for Kamiwaza 1.0.0

This is documentation for Kamiwaza 1.0.0, which is no longer actively maintained. For the current GA release, see 1.0.1.

Overview

Kamiwaza uses S3-compatible object storage for two platform workflows:

• workroom context file persistence
• Skills Library package import and download

If object storage is not configured, file-backed user workflows can fail with errors such as:

Workroom storage is not configured for Skills Library.

Use this guide when your deployment stores workroom content in external AWS S3 instead of the default in-cluster object storage.

When You Need This

Configure AWS S3 if your workroom content must live in an external AWS S3 bucket and you want the following to work reliably:

• context file upload and persistence
• Skills Library package import
• Skills Library package download

Prerequisites

Before you start, make sure you have:

• an AWS S3 bucket
• an IAM user or role with access to that bucket
• kubectl access to the target cluster
• a values override file for the deployment configuration

The IAM identity should be allowed to perform at least:

• s3:ListBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject

Choose a Configuration Surface

There are two ways to configure external S3 workroom storage, depending on how you deploy:

Configuration Model

Under the hood, the core chart exposes a single object-storage configuration block. The storage platform fills it in from storage.workrooms.*; direct chart consumers set it themselves:

context:

objectStorage:

enabled: true

defaultBucket: ""

defaultRegion: ""

defaultPrefix: "context/raw"

endpointUrl: ""

credentialsSecretRef:

name: ""

accessKeyIdKey: "access_key_id"

secretAccessKeyKey: "secret_access_key"

sessionTokenKey: "session_token"

Option 1: Static AWS Access Keys

Use this option when your cluster does not already provide AWS credentials to the pods.

1. Create a Kubernetes Secret

Core pods wait on this Secret, so create it before deploying:

kubectl create namespace kamiwaza --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic core-s3 \

-n kamiwaza \

--from-literal=access_key_id="<aws-access-key-id>" \

--from-literal=secret_access_key="<aws-secret-access-key>"

If you are using temporary session credentials:

kubectl create secret generic core-s3 \

-n kamiwaza \

--from-literal=access_key_id="<aws-access-key-id>" \

--from-literal=secret_access_key="<aws-secret-access-key>" \

--from-literal=session_token="<aws-session-token>"

2a. Helmfile installs: configure storage.workrooms (preferred)

Add this to deploy/cluster/values/storage-overrides.yaml:

storage:

workrooms:

backend: s3

s3:

region: "us-west-2"

bucket: "my-kamiwaza-artifacts"

prefix: "context/raw"

endpoint: ""

existingSecret: "core-s3"

Option 2: Ambient AWS Credentials

Use this option if your cluster already provides AWS credentials to pods through IAM roles or another AWS-native mechanism. In this case, do not create a Secret.

Helmfile installs

Add this to deploy/cluster/values/storage-overrides.yaml:

storage:

workrooms:

backend: s3

s3:

region: "us-west-2"

bucket: "my-kamiwaza-artifacts"

prefix: "context/raw"

endpoint: ""

existingSecret: ""

Restart Existing Deployments

If the cluster is already running, restart the core workloads after applying the updated config:

kubectl rollout restart deployment/core-scheduler -n kamiwaza

kubectl delete pod -n kamiwaza -l ray.io/node-type=head

kubectl delete pod -n kamiwaza -l ray.io/node-type=worker

Verify the Configuration

Check the ConfigMap

Verify that the non-secret S3 settings are present:

kubectl get configmap core-config -n kamiwaza -o yaml | grep CONTEXT_SERVICE_S3

Troubleshooting

Error: secret "core-s3" not found (CreateContainerConfigError)

Core pods (core-scheduler, core-raycluster-head) are stuck in CreateContainerConfigError. This means core was configured to read S3 credentials from a Secret that does not exist in the kamiwaza namespace.

Fix:

1. If you intended the default RGW storage: remove the core.context.objectStorage block from your overrides and re-apply the release.
2. If you intended external S3: create the Secret (Option 1, step 1).

Security Notes

• Do not store AWS secrets in workroom attributes or user-editable metadata.
• Prefer Kubernetes Secrets or ambient AWS identity over plain-text credentials in values files.
• Leave endpointUrl empty for AWS S3. Set it only when using a non-AWS S3-compatible service.