# Version: 1.0.1 (Latest)

Kamiwaza uses S3-compatible object storage for two platform workflows:

- workroom context file persistence
- Skills Library package import and download

If object storage is not configured, file-backed user workflows can fail with errors such as:

```text
Workroom storage is not configured for Skills Library.
```

Use this guide when your deployment stores workroom content in **external AWS S3** instead of the default in-cluster object storage.  
Default RGW installs need no configuration — do not apply this page to them.

## When You Need This
Configure AWS S3 if your workroom content must live in an external AWS S3 bucket and you want the following to work reliably:

- context file upload and persistence
- Skills Library package import
- Skills Library package download

If you are on the default `rook-rgw` storage lane and have no requirement to move workroom content to AWS, you do not need this page.

## Prerequisites
Before you start, make sure you have:

- an AWS S3 bucket
- an IAM user or role with access to that bucket
- `kubectl` access to the target cluster
- a values override file for the deployment configuration

The IAM identity should be allowed to perform at least:

- `s3:ListBucket`
- `s3:GetObject`
- `s3:PutObject`
- `s3:DeleteObject`

## Choose a Configuration Surface
There are two ways to configure external S3 workroom storage, depending on how you deploy:

| Deployment | Configure via |
| --- | --- |
| `deploy` repo Helmfile install (v1.0 storage platform) | `storage.workrooms` in `deploy/cluster/values/storage-overrides.yaml` — **preferred** |
| Direct `core` chart values (no umbrella chart / no Helmfile) | `context.objectStorage` in your core values file |

On Helmfile installs, the storage platform materializes `storage.workrooms.*` into the core chart's `context.objectStorage` block for you. This applies to `install-prod.sh` installs too: they deploy through the same Helmfile environments, so a storage block in `cluster/values/storage-overrides.yaml` is honored there as well.

Do not use `deploy/cluster/values/overrides.yaml` for storage configuration. It is a late umbrella-chart override that takes precedence over the storage platform's automatic wiring. Deployments configured on 0.13.x used `overrides.yaml` for this purpose; when upgrading to a storage-platform release, migrate the block to `storage-overrides.yaml` — or remove it entirely if you are returning to the default RGW lane.

## Configuration Model
Under the hood, the `core` chart exposes a single object-storage configuration block. The storage platform fills it in from `storage.workrooms.*`; direct chart consumers set it themselves:

```yaml
context:
  objectStorage:
    enabled: true
    defaultBucket: ""
    defaultRegion: ""
    defaultPrefix: "context/raw"
    endpointUrl: ""
    credentialsSecretRef:
      name: ""
      accessKeyIdKey: "access_key_id"
      secretAccessKeyKey: "secret_access_key"
      sessionTokenKey: "session_token"
```

When `enabled` is omitted, the chart infers it. This configuration renders into:

- the `core-config` ConfigMap for non-secret S3 settings
- the `core-scheduler` deployment for secret-backed AWS credentials
- the Ray head and worker pods for the same credentials
- the storage-janitor CronJob

## Option 1: Static AWS Access Keys
Use this option when your cluster does not already provide AWS credentials to the pods.
### 1. Create a Kubernetes Secret
Core pods wait on this Secret, so create it before deploying:

```bash
kubectl create namespace kamiwaza --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic core-s3 \
  -n kamiwaza \
  --from-literal=access_key_id="<aws-access-key-id>" \
  --from-literal=secret_access_key="<aws-secret-access-key>"
```

If you are using temporary session credentials, include the session token:

```bash
kubectl create secret generic core-s3 \
  -n kamiwaza \
  --from-literal=access_key_id="<aws-access-key-id>" \
  --from-literal=secret_access_key="<aws-secret-access-key>" \
  --from-literal=session_token="<aws-session-token>"
```

### 2a. Helmfile installs: configure `storage.workrooms` (preferred)
Add this to `deploy/cluster/values/storage-overrides.yaml`:

```yaml
storage:
  workrooms:
    backend: s3
    s3:
      region: "us-west-2"
      bucket: "my-kamiwaza-artifacts"
      prefix: "context/raw"
      endpoint: ""
      existingSecret: "core-s3"
```

### 3. Apply the Updated Configuration
- Deploy-repo Helmfile installs: re-run your usual deploy target — `make install`.
- Direct chart consumers: `helm upgrade --install <release> <chart> -n kamiwaza -f <your-values>.yaml`.

## Option 2: Ambient AWS Credentials
Use this option if your cluster already provides AWS credentials to pods through IAM roles or another AWS-native mechanism.
### 1. Configure with Ambient Credentials
On Helmfile installs, add this to `deploy/cluster/values/storage-overrides.yaml`:

```yaml
storage:
  workrooms:
    backend: s3
    s3:
      region: "us-west-2"
      bucket: "my-kamiwaza-artifacts"
      prefix: "context/raw"
      endpoint: ""
      existingSecret: ""
```

Confirm that `enabled` is **required** here — with an empty Secret name the chart cannot infer it.

## Restart Existing Deployments
If the cluster is already running, restart the core workloads after applying the updated config:

```bash
kubectl rollout restart deployment/core-scheduler -n kamiwaza

kubectl delete pod -n kamiwaza -l ray.io/node-type=head

kubectl delete pod -n kamiwaza -l ray.io/node-type=worker
```

## Verify the Configuration
### Check the ConfigMap
Verify that the non-secret S3 settings are present:

```bash
kubectl get configmap core-config -n kamiwaza -o yaml | grep CONTEXT_SERVICE_S3
```

### Check the Scheduler
Verify the rendered credential reference without echoing any secret values:

```bash
kubectl get deploy core-scheduler -n kamiwaza -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID
```

### Check the Ray Head Pod
Verify the Ray head pod carries the same configuration:

```bash
kubectl get pod -n kamiwaza -l ray.io/node-type=head -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID
```

### Validate in the Product
Retry a workflow that depends on workroom storage:
- import a Skills Library package
- upload a context file

## Troubleshooting
### Error: secret "core-s3" not found
Core pods are stuck in `CreateContainerConfigError`. This means core was configured to read S3 credentials from a Secret that does not exist in the `kamiwaza` namespace. Check which Secret and key names core actually references:

```bash
kubectl get deploy core-scheduler -n kamiwaza -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID
```

### Error: Workroom storage is not configured for Skills Library
This usually means core does not see `CONTEXT_SERVICE_S3_DEFAULT_BUCKET`. Check your Helm values file includes `context.objectStorage.defaultBucket`.

### Error: Unable to locate credentials
Ensure the Secret exists in the `kamiwaza` namespace and the credential reference matches the Secret name.

### Error: AccessDenied or SignatureDoesNotMatch
This usually means the credentials are present but not valid for the target bucket or region.