AWS S3 Workroom Storage | Kamiwaza Docs

Version: 1.0.1 (Latest)

Kamiwaza uses S3-compatible object storage for two platform workflows:

If object storage is not configured, file-backed user workflows can fail with errors such as:

Workroom storage is not configured for Skills Library.

Use this guide when your deployment stores workroom content in external AWS S3 instead of the default in-cluster object storage.
Default RGW installs need no configuration — do not apply this page to them.

When You Need This

Configure AWS S3 if your workroom content must live in an external AWS S3 bucket and you want the following to work reliably:

If you are on the default rook-rgw storage lane and have no requirement to move workroom content to AWS, you do not need this page.

Prerequisites

Before you start, make sure you have:

The IAM identity should be allowed to perform at least:

Choose a Configuration Surface

There are two ways to configure external S3 workroom storage, depending on how you deploy:

Deployment Configure via
deploy repo Helmfile install (v1.0 storage platform) storage.workrooms in deploy/cluster/values/storage-overrides.yaml — preferred
Direct core chart values (no umbrella chart / no Helmfile) context.objectStorage in your core values file

On Helmfile installs, the storage platform materializes storage.workrooms.* into the core chart's context.objectStorage block for you. This applies to install-prod.sh installs too: they deploy through the same Helmfile environments, so a storage block in cluster/values/storage-overrides.yaml is honored there as well.

Do not use deploy/cluster/values/overrides.yaml for storage configuration. It is a late umbrella-chart override that takes precedence over the storage platform's automatic wiring. Deployments configured on 0.13.x used overrides.yaml for this purpose; when upgrading to a storage-platform release, migrate the block to storage-overrides.yaml — or remove it entirely if you are returning to the default RGW lane.

Configuration Model

Under the hood, the core chart exposes a single object-storage configuration block. The storage platform fills it in from storage.workrooms.*; direct chart consumers set it themselves:

context:
  objectStorage:
    enabled: true
    defaultBucket: ""
    defaultRegion: ""
    defaultPrefix: "context/raw"
    endpointUrl: ""
    credentialsSecretRef:
      name: ""
      accessKeyIdKey: "access_key_id"
      secretAccessKeyKey: "secret_access_key"
      sessionTokenKey: "session_token"

When enabled is omitted, the chart infers it. This configuration renders into:

Option 1: Static AWS Access Keys

Use this option when your cluster does not already provide AWS credentials to the pods.

1. Create a Kubernetes Secret

Core pods wait on this Secret, so create it before deploying:

kubectl create namespace kamiwaza --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic core-s3 \
  -n kamiwaza \
  --from-literal=access_key_id="<aws-access-key-id>" \
  --from-literal=secret_access_key="<aws-secret-access-key>"

If you are using temporary session credentials, include the session token:

kubectl create secret generic core-s3 \
  -n kamiwaza \
  --from-literal=access_key_id="<aws-access-key-id>" \
  --from-literal=secret_access_key="<aws-secret-access-key>" \
  --from-literal=session_token="<aws-session-token>"

2a. Helmfile installs: configure storage.workrooms (preferred)

Add this to deploy/cluster/values/storage-overrides.yaml:

storage:
  workrooms:
    backend: s3
    s3:
      region: "us-west-2"
      bucket: "my-kamiwaza-artifacts"
      prefix: "context/raw"
      endpoint: ""
      existingSecret: "core-s3"

3. Apply the Updated Configuration

Option 2: Ambient AWS Credentials

Use this option if your cluster already provides AWS credentials to pods through IAM roles or another AWS-native mechanism.

1. Configure with Ambient Credentials

On Helmfile installs, add this to deploy/cluster/values/storage-overrides.yaml:

storage:
  workrooms:
    backend: s3
    s3:
      region: "us-west-2"
      bucket: "my-kamiwaza-artifacts"
      prefix: "context/raw"
      endpoint: ""
      existingSecret: ""

Confirm that enabled is required here — with an empty Secret name the chart cannot infer it.

Restart Existing Deployments

If the cluster is already running, restart the core workloads after applying the updated config:

kubectl rollout restart deployment/core-scheduler -n kamiwaza

kubectl delete pod -n kamiwaza -l ray.io/node-type=head

kubectl delete pod -n kamiwaza -l ray.io/node-type=worker

Verify the Configuration

Check the ConfigMap

Verify that the non-secret S3 settings are present:

kubectl get configmap core-config -n kamiwaza -o yaml | grep CONTEXT_SERVICE_S3

Check the Scheduler

Verify the rendered credential reference without echoing any secret values:

kubectl get deploy core-scheduler -n kamiwaza -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID

Check the Ray Head Pod

Verify the Ray head pod carries the same configuration:

kubectl get pod -n kamiwaza -l ray.io/node-type=head -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID

Validate in the Product

Retry a workflow that depends on workroom storage:

Troubleshooting

Error: secret "core-s3" not found

Core pods are stuck in CreateContainerConfigError. This means core was configured to read S3 credentials from a Secret that does not exist in the kamiwaza namespace. Check which Secret and key names core actually references:

kubectl get deploy core-scheduler -n kamiwaza -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID

Error: Workroom storage is not configured for Skills Library

This usually means core does not see CONTEXT_SERVICE_S3_DEFAULT_BUCKET. Check your Helm values file includes context.objectStorage.defaultBucket.

Error: Unable to locate credentials

Ensure the Secret exists in the kamiwaza namespace and the credential reference matches the Secret name.

Error: AccessDenied or SignatureDoesNotMatch

This usually means the credentials are present but not valid for the target bucket or region.