# Documentation for Kamiwaza 0.8.1

This document provides comprehensive data ingestion functionality for the Kamiwaza AI Platform. Located in `kamiwaza_client/services/ingestion.py`, this service handles data ingestion workflows, dataset processing, and document handling with embedding capabilities.

## Key Features

- **Data Ingestion**
- **Dataset Catalog Integration**
- **Document Processing**
- **Embedding Generation**
- **Batch Processing Support**

## Data Ingestion

### Available Methods

- `ingest(data: Union[str, List[str], Dict[str, Any]], **kwargs) -> IngestionResponse`: Ingest data
- `ingest_dataset(dataset: Dataset, **kwargs) -> DatasetIngestionResponse`: Ingest dataset to catalog
- `initialize_embedder(provider: str = "default", **kwargs) -> None`: Initialize embedding provider
- `process_documents(documents: List[Document], **kwargs) -> ProcessingResponse`: Process documents

```python
# Simple data ingestion

response = client.ingestion.ingest(
    data="Sample text data",
    chunk_size=512
)

# Dataset ingestion

response = client.ingestion.ingest_dataset(
    dataset=dataset_obj,
    embedding_config={
        "provider": "huggingface",
        "model": "sentence-transformers/all-mpnet-base-v2"
    }
)

# Initialize embedder

client.ingestion.initialize_embedder(
    provider="huggingface",
    model_name="sentence-transformers/all-mpnet-base-v2"
)

# Process documents

response = client.ingestion.process_documents(
    documents=[
        Document(text="doc1", metadata={"source": "file1"}),
        Document(text="doc2", metadata={"source": "file2"})
    ],
    chunk_size=512,
    overlap=50
)
```

## Integration with Other Services

The Ingestion Service works in conjunction with:

1. **Embedding Service** - For generating embeddings of ingested text
2. **VectorDB Service** - For storing processed vectors
3. **Catalog Service** - For dataset management
4. **Retrieval Service** - For accessing processed documents

## Error Handling

The service includes built-in error handling for common scenarios:

```python
try:
    response = client.ingestion.ingest(data)
except EmbeddingError:
    print("Embedding generation failed")
except VectorDBError:
    print("Vector storage failed")
except ProcessingError as e:
    print(f"Document processing failed: {e}")
except APIError as e:
    print(f"Operation failed: {e}")
```

## Best Practices

1. **Initialize embedder before ingestion**
2. **Use appropriate chunk sizes**
3. **Include relevant metadata**
4. **Process documents in batches**
5. **Monitor ingestion progress**
6. **Handle errors appropriately**
7. **Clean up failed ingestions**
8. **Validate data before ingestion**

## Performance Considerations

- Batch size affects processing speed
- Embedding generation time
- Vector database insertion overhead
- Memory usage during processing
- Network bandwidth for large datasets

## Data Formats

The service supports various input formats:

1. **Raw Text**
   - Single strings
   - Lists of strings
2. **Structured Data**
   - JSON objects
   - Dictionaries
3. **Documents**
   - Custom Document objects
   - Metadata support
4. **Datasets**
   - Catalog integration
   - Batch processing