Help & Fixes | Kamiwaza Docs

Getting Help

If you have questions or run into issues, we're here to help:

• Join our Discord community
• Visit our website
• Visit our repo
• Try our client SDK
• Contact our support team

We're committed to making your experience with Kamiwaza as smooth as possible.

Reporting Issues

When reporting issues to our support team or community, please include:

• Environment Details: OS version, Kubernetes distribution and version, hardware specs, and the Kamiwaza release version
• Error Messages: Complete error text and stack traces
• Steps to Reproduce: Detailed steps that led to the issue
• Logs: Relevant Kubernetes pod logs, platform logs, and UI error details
• Configuration: Any custom configuration or settings

This information helps us provide faster and more accurate solutions to your problems.

Common Issues and Fixes

Installation Issues

Docker GPU Error: Could Not Select Device Driver

Problem: NVIDIA Container Runtime not found or misconfigured.

Solution:

• Ensure NVIDIA drivers are properly installed
• Install NVIDIA Container Toolkit
• Verify Docker can access GPU devices

Port Already in Use

Problem: Kamiwaza fails to start because required ports are occupied.

Solution:

• Check what's running on ports 3000, 8000, 5432, 19530, 9090
• Stop conflicting services or change Kamiwaza's port configuration
• Use lsof -i :PORT_NUMBER to identify processes using specific ports

Insufficient System Resources

Problem: Installation fails due to low disk space, RAM, or CPU cores.

Solution:

• Ensure at least 16GB RAM available
• Verify CPU supports required virtualization features

Model Deployment Issues

Gated Model Downloads and Rate Limits

Problem: Downloads fail for gated models (Llama, Mistral, etc.) or you encounter Hugging Face rate limit errors.

Symptoms:

• "Access denied" or "401 Unauthorized" when downloading certain models
• "Rate limit exceeded" errors during model downloads
• Gated models appear in search but fail to download

Solution: Add your Hugging Face token to your Kamiwaza deployment configuration:

1. Get a Hugging Face token from huggingface.co/settings/tokens (a read token is sufficient)
2. For gated models, accept the license terms on the model's Hugging Face page
3. Add the token to the Secret or deployment values used for your Kamiwaza environment:

   HF_TOKEN="hf_your_token_here"

4. Apply the updated configuration using your standard cluster release workflow and confirm the model-serving pods restart successfully.
5. If you need setup details for secrets or deployment values, refer to the Configuration Reference and the Administrator Guide.

Model Deployment Failures

Problem: Models fail to deploy or become unavailable.

Solutions:

• Model not found: Ensure the model exists in your catalog or use Novice Mode
• Checkpoint too large for VRAM: Choose a smaller/quantized variant (AWQ, MLX, GGUF) or reduce batch size
• Service unavailable/port errors: Stop/Remove and redeploy the model
• Outdated catalog: Refresh the Models page or restart the server

Performance Problems

Problem: Slow responses or high resource usage.

Solutions:

• Slow responses: Use faster models or quantized variants; reduce max tokens and context length
• High memory/OOM: Lower batch size, context length, and KV cache; use lower-VRAM variants
• Cold starts: First request may be slower; send a short warm-up prompt after deploy

SDK and API Issues

Module Import Error

Problem: ModuleNotFoundError: No module named 'kamiwaza_sdk' when using notebooks and Kamiwaza SDK.

Solution:

!pip uninstall -y kamiwaza kamiwaza-sdk

!pip install kamiwaza-sdk

Then restart the notebook kernel or Python session before importing kamiwaza_sdk again.

App Garden Issues

App Not Showing Latest Version

Problem: An app in App Garden isn't displaying the latest content or updates due to cache TTL.

Solution: Force a cache refresh by syncing remote templates:

# Step 1: Get an auth token

TOKEN=$(curl -sk -X POST "https://your-kamiwaza-instance/api/auth/token" \

-H "Content-Type: application/x-www-form-urlencoded" \

-d "username=admin&password=your-password" \

| jq -r '.access_token')

# Step 2: Force a cache refresh

curl -sk -X POST "https://your-kamiwaza-instance/api/v1/apps/{app_id}/refresh" \

-H "Authorization: Bearer ${TOKEN}"

General Troubleshooting Steps

When encountering issues, follow these diagnostic steps:

1. Check Service Status: Verify all Kamiwaza services are running
2. Review Logs: Check container logs for specific error messages
3. Verify Resources: Ensure sufficient CPU, RAM, and disk space
4. Test Connectivity: Verify network connectivity between components
5. Restart Services: Try stopping and restarting affected services
6. Check Configuration: Verify configuration files and environment variables