Base System Requirements

Supported Operating Systems & Architecture

• Linux:
  • Ubuntu 24.04 and 22.04 LTS via .deb package installation (x64/amd64 architecture only)
  • Redhat Enterprise Linux (RHEL) 9
• Windows: 11 (x64 architecture) via WSL with MSI installer
• macOS: 12.0 or later, Apple Silicon (ARM64) only (community edition only)

CPU Requirements

• Architecture:
  • Linux: x64/amd64 (64-bit)
  • macOS: ARM64 (Apple Silicon) only
  • Windows: x64 (64-bit)
• Minimum Cores: 8+ cores
• Recommended Cores: 16+ cores for CPU-based inference workloads

Core Software Requirements

• Python: Python 3.10 for tarball installations; Python 3.12 for .deb/.msi installations
• Docker: Docker Engine with Compose v2
• Node.js: 22.x (installed via NVM during setup)
• Browser: Chrome Version 141+ (tested and recommended)
• GPU Support: NVIDIA GPU with compute capability 7.0+ (Linux only) or NVIDIA RTX/Intel Arc (Windows via WSL)

Memory Requirements

System RAM

• Minimum: 16GB RAM
• Recommended: 32GB+ RAM for CPU-based inference workloads
• GPU Workloads: 16GB+ system RAM (32GB+ recommended)

GPU Memory (vRAM)

• GPU Inference: 16GB+ vRAM required
• Recommended: 32GB+ vRAM for optimal GPU inference performance

Windows (WSL-based) Specific

• Minimum: 16GB RAM
• Recommended: 32GB+ RAM
• Memory Allocation: 50-75% of system RAM dedicated to Kamiwaza during installation

Storage Requirements

Storage Performance

• Required: SSD (Solid State Drive)
• Preferred: NVMe SSD for optimal performance
• Minimum: SATA SSD
• Note: Models weights can be on a separate HDD but loads time will increase

Storage Capacity

Linux/macOS

• Minimum: 100GB free disk space
• Recommended: 200GB+ free disk space
• Enterprise Edition: Additional space for /opt/kamiwaza persistence

Windows

• Minimum: 100GB free disk space
• Recommended: 200GB+ free space on SSD
• WSL: Automatically manages Ubuntu 24.04 installation space

Hardware Recommendation Tiers

Kamiwaza is a distributed AI platform built on Ray that supports both CPU-only and GPU-accelerated inference. Hardware requirements vary significantly based on:

• Model size: From 0.6B to 70B+ parameters
• Deployment scale: Single-node development vs multi-node production
• Inference engine: LlamaCpp (CPU/GPU), VLLM (GPU), MLX (Apple Silicon)
• Workload type: Interactive chat, batch processing, RAG pipelines

GPU Memory Requirements by Model Size

The table below provides real-world GPU memory requirement estimates for representative models at different scales. These estimates assume FP8 and include overhead for context windows and batch processing.

Tier 1: Development & Small Models

Use Case: Local development, testing, small to medium model deployment (up to 13B parameters)

Hardware Specifications:

• CPU: 8-16 cores / 16-32 threads
• RAM: 32GB (16GB minimum)
• Storage: 200GB NVMe SSD (100GB minimum)
• GPU: Optional - Single GPU with 16-24GB VRAM
  • NVIDIA RTX 4090 (24GB)
  • NVIDIA RTX 4080 (16GB)
  • NVIDIA T4 (16GB)
• Network: 1-10 Gbps

Workload Capacity:

• Low-volume workloads: 1-10 concurrent requests (supports dozens of interactive users)
• Development, testing, and proof-of-concept deployments
• Light production workloads

Tier 2: Production - Medium to Large Models

Use Case: Production deployment of medium to large models (13B-70B parameters), high throughput

Hardware Specifications:

• CPU: 32 cores / 64 threads
• RAM: 128-256GB system RAM
• Storage: 1-2TB NVMe SSD
• GPU: 1-4 GPUs with 40GB+ VRAM each
  • 1-4x NVIDIA B200 (192GB HBM3e)
  • 1-4x NVIDIA H200 (141GB HBM3e)
  • 1-4x NVIDIA RTX 6000 Pro Blackwell (48GB)
  • 1-2x NVIDIA H100 (80GB)
  • 1-4x NVIDIA A100 (40GB or 80GB)
  • 1-2x NVIDIA L40S (48GB)
  • 2-4x NVIDIA A10G (24GB) for tensor parallelism
• Network: 25-40 Gbps

Workload Capacity:

• Medium-scale production: 100s to 1,000+ concurrent requests (supports thousands of interactive users)
• Example: Per-GPU batch size of 32 across 8 GPUs = 256 concurrent requests; batch size of 128 = 1,024 requests
• Production chat applications
• Complex RAG pipelines with embedding generation
• Batch inference

Tier 3: Enterprise Multi-Node Cluster

Use Case: Enterprise deployment with multiple models, high availability, horizontal scaling, 99.9%+ SLA

Cluster Architecture:

Head Node (Control Plane):

• CPU: 16 cores / 32 threads
• RAM: 64GB
• Storage: 500GB NVMe SSD
• GPU: Same class as worker nodes (homogeneous cluster recommended)
• Role: Ray head, API gateway, scheduling, monitoring (head performs minimal extra work; Ray backend load is distributed across nodes)

Worker Nodes (3+ nodes for HA):

• CPU: 32-64 cores / 64-128 threads per node
• RAM: 256-512GB per node
• Storage: 2TB NVMe SSD per node (local cache)
• GPU: 4-8 GPUs per node (same class as head node)
• Network: 40-100 Gbps (InfiniBand for HPC workloads)

Shared Storage:

• High-performance NAS or distributed filesystem (Lustre, CephFS)
• 10TB+ capacity, NVMe-backed
• 10+ GB/s aggregate sequential throughput
• Low-latency access (< 5ms) from all nodes

Workload Capacity:

• Multiple models deployed simultaneously
• High-scale production: 1,000–10,000+ concurrent requests (supports tens of thousands of interactive users)
• Batch sizes scale with GPU count and model size; smaller requests enable higher throughput per GPU
• High availability with automatic failover
• Horizontal auto-scaling based on load
• Production SLAs (99.9% uptime)

Cloud Provider Instance Mapping

AWS EC2 Instance Types

Google Cloud Platform (GCP) Instance Types

Microsoft Azure Instance Types

Windows-Specific Prerequisites

• Windows Subsystem for Linux (WSL) installed and enabled
• Administrator access required for initial setup
• Windows Terminal (recommended for optimal WSL experience)

Dependencies & Components

Required System Packages

See platform-specific installation instructions

NVIDIA Components (Linux GPU Support)

• NVIDIA Driver (550-server recommended)
• NVIDIA Container Toolkit
• nvidia-docker2

Windows Components (Automated via MSI Installer)

• Windows Subsystem for Linux (WSL 2)
• Ubuntu 24.04 LTS (automatically downloaded and configured)
• Docker Engine (configured within WSL)
• GPU drivers and runtime (automatically detected and configured)
• Node.js 22 (via NVM within WSL environment)

Docker Configuration Requirements

• Docker Engine with Compose v2
• User must be in docker group
• Swarm mode (Enterprise Edition)
• Docker data root configuration (configurable)

Required Directory Structure

Enterprise Edition

Note this is created by the installer and present in cloud marketplace images.

/etc/kamiwaza/

├── config/

├── ssl/      # Cluster certificates

└── swarm/    # Swarm tokens

/opt/kamiwaza/

├── containers/  # Docker root (configurable)

├── logs/

├── nvm/        # Node Version Manager

└── runtime/    # Runtime files

Community Edition

We recommend ${HOME}/kamiwaza or something similar for KAMIWAZA_ROOT.

$KAMIWAZA_ROOT/

├── env.sh

├── runtime/

└── logs/

Network Configuration

Network Bandwidth Requirements

Single Node Deployment

Network Bandwidth:

• Minimum: 1 Gbps (for model downloads, API traffic)
• Recommended: 10 Gbps (for high-throughput inference)

Multi-Node Cluster

Inter-Node Network:

• Minimum: 10 Gbps Ethernet
• Recommended: 25-40 Gbps Ethernet or InfiniBand
• Latency: < 1ms between nodes (same datacenter/availability zone)

Required Kernel Modules (Enterprise Edition Linux Only)

Required modules for Swarm container networking:

• overlay
• br_netfilter

System Network Parameters (Enterprise Edition Linux Only)

These will be set by the installer.

# Required sysctl settings for Swarm networking

net.bridge.bridge-nf-call-iptables  = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.ipv4.ip_forward                 = 1

Community Edition Networking

• Uses standard Docker bridge networks
• No special kernel modules or sysctl settings required
• Simplified single-node networking configuration

Detailed Storage Requirements

Capacity Planning

Storage Performance Requirements

Local Storage (Single Node)

Storage Type:

• Minimum: SATA SSD (500 MB/s sequential read)
• Recommended: NVMe SSD (2000+ MB/s sequential read)
• Note: HDD: Only recommended for non-dynamic model loads and low KV cache usage - model load times can be very long (15+ minutes); models are in memory after load

Performance Targets:

• Sequential Read: 2000+ MB/s (model loading)
• Sequential Write: 1000+ MB/s (model downloads, checkpoints)
• 4K Random Read IOPS: 50,000+ (database, concurrent access)
• 4K Random Write IOPS: 20,000+ (database writes, logs)

Shared Storage (Multi-Node Clusters)

Network Filesystem Requirements:

• Protocol: NFSv4, Lustre, CephFS, or S3-compatible object storage
• Network Bandwidth: 10 Gbps minimum, 40+ Gbps for production
• Network Latency: < 5ms between nodes and storage
• Sequential Throughput: 5+ GB/s aggregate (10+ GB/s for large clusters)