This is documentation for Kamiwaza **1.0.0**, which is no longer actively maintained. For the current GA release, see [**1.0.1**](https://docs.kamiwaza.ai/).

Version: 1.0.0

Submit and manage Ray jobs on local or remote clusters. Jobs run Python entrypoints on the cluster's Ray runtime and can return structured results via a log marker protocol.

## Prerequisites

- Admin authentication (JWT token with admin role)
- Ray cluster running on the target cluster

## Submit a Job (Async)

```bash
curl -sk -X POST "https://kamiwaza.test/api/cluster/jobs/submit" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python train.py --epochs 10",
    "runtime_env": {"pip": ["numpy", "pandas"]},
    "timeout_seconds": 300
  }'
```

Response:

```json
{"job_id": "uuid", "ray_job_id": "raysubmit_...", "status": "PENDING"}
```

## Run a Job (Synchronous)

Submits and blocks until completion, returning the extracted result:

```bash
curl -sk -X POST "https://kamiwaza.test/api/cluster/jobs/run" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python inference.py",
    "timeout_seconds": 60
  }'
```

Response:

```json
{
  "job_id": "uuid",
  "ray_job_id": "raysubmit_...",
  "status": "SUCCEEDED",
  "result": {"accuracy": 0.95},
  "duration_seconds": 12.3
}
```

## Check Job Status

```bash
curl -sk "https://kamiwaza.test/api/cluster/jobs/{job_id}/status" \
  -H "Authorization: Bearer $TOKEN"
```

Returns the full job record including status, timestamps, and error details if failed.

## Get Job Logs

```bash
curl -sk "https://kamiwaza.test/api/cluster/jobs/{job_id}/logs" \
  -H "Authorization: Bearer $TOKEN"
```

Returns Ray stdout/stderr for the job.

## Extract Job Result

```bash
curl -sk "https://kamiwaza.test/api/cluster/jobs/{job_id}/result" \
  -H "Authorization: Bearer $TOKEN"
```

Extracts structured JSON from the job's stdout. The job code must print a result marker:

```python
import json

result = {"accuracy": 0.95, "model": "v2"}

print(f"KZ_MESH_RUN_ON_JSON::{json.dumps(result)}")
```

| Status Code | Meaning |
| --- | --- |
| 200 | Result extracted successfully |
| 409 | Job has not succeeded yet |
| 410 | Ray logs expired (result no longer available) |

## Cancel a Job

```bash
curl -sk -X POST "https://kamiwaza.test/api/cluster/jobs/{job_id}/cancel" \
  -H "Authorization: Bearer $TOKEN"
```

Requests cancellation of a running job. The job transitions to STOPPED status.

## Job Lifecycle

| Status | Description |
| --- | --- |
| PENDING | Submitted to Ray, not yet running |
| RUNNING | Executing on Ray cluster |
| SUCCEEDED | Completed successfully |
| FAILED | Exited with error |
| STOPPED | Cancelled by user or timed out |

## Timeout and Auto-Cancel

Set `timeout_seconds` on submission. If the job exceeds this duration during a `/run` call, it is automatically cancelled and marked with `timed_out: true`.

## API Reference

| Method | Path | Description |
| --- | --- | --- |
| POST | `/cluster/jobs/submit` | Async submission |
| POST | `/cluster/jobs/run` | Sync submit + poll + result |
| GET | `/cluster/jobs/{id}/status` | Job status with Ray refresh |
| GET | `/cluster/jobs/{id}/result` | Extract structured result |
| GET | `/cluster/jobs/{id}/logs` | Ray stdout/stderr |
| POST | `/cluster/jobs/{id}/cancel` | Cancel running job |