Distributed Data Engine | Kamiwaza Docs

Version: 1.0.1 (Latest)

Kamiwaza’s Distributed Data Engine (DDE) aligns unstructured and tabular content under a unified ingestion framework. Connector-driven pipelines push cleaned documents into the platform’s vector stores while keeping credentials and scheduling consistent with the rest of the control plane. Subsequent point releases continue to build on the same structure, so the guidance below applies to the current GA build unless otherwise noted.

DDE stands for Distributed Data Engine.

Platform assumptions

Connector workflow

  1. Create a connector – Choose the source system (S3, SharePoint, file upload, and so on) and supply the required fields through the UI or supported API.
  2. Secure credentials – Prefer referencing an existing Kamiwaza secret. Secrets are encrypted at rest and can be rotated without recreating the connector.
  3. Schedule ingestion – Select one-time or recurring runs. DDE batches updates to minimize load on the source system.
  4. Monitor jobs – Each run emits status events and logging metadata that flow into the standard Kamiwaza logging and observability paths.

Supported sources

Source Notes
File Files accessible to the deployment through supported upload or storage paths.
Amazon S3 Uses access key/secret with read permission on the target bucket/prefix.
Kafka Streams and batch pulls for topic-backed ingestion.
Postgres Reads structured data for catalog and retrieval.
Hive Reads warehouse data for catalog and retrieval.
Slack Pulls channel content for indexing and retrieval.

Need a connector that isn’t listed? Contact Kamiwaza Support to discuss roadmap status or professional-services extensions.

Ingestion service APIs (scheduling + runs)

The ingestion service provides job-based ingestion and connector execution. These endpoints are available behind the standard API gateway:

Each ingestion request specifies a source_type (for example s3 or postgres) and a kwargs/conn_args payload that holds the connector-specific configuration.

DDE connector and document APIs

DDE connector and document endpoints are mounted under the ingestion service with /api/dde/... paths. These are used by the UI and by automated ingestion workflows:

Connectors carry security metadata such as system_high (the maximum classification allowed) and an optional default_security_marking applied when documents lack explicit markings.

Audio ingest

Audio files submitted through the context manager are routed to the platform's OmniParse transcription endpoint before indexing. The context manager's supported media extensions are .aac, .aif, .aiff, .flac, .m4a, .m4b, .mp3, .oga, .ogg, .opus, .wav, .wma, plus .mov, .mp4, and .webm when only the audio track needs to be transcribed.

Transcribed text enters the same extraction and indexing path as documents, so media sources appear in search and retrieval with the same security markings as their parent connector. No connector-side configuration change is required; the routing is automatic based on file type when OmniParse is configured. The automatic OmniParse path enables use_omniparse=true with strict_omniparse=false in the underlying context manager. That means OmniParse failures fall back to the built-in extractors where one exists, instead of failing the whole job immediately. For audio/video files themselves, successful transcription is still required to produce chunks, so empty transcription results behave like empty content.

Current limits come from the context manager rather than a DDE-specific knob: the default decoded file-size limit is 100 MB (configurable via CONTEXT_MANAGER_MAX_FILE_SIZE), the streaming upload ceiling is 200 MB, and the OmniParse transcription timeout defaults to 300 seconds.

For detailed documentation on OmniParse deployment, capabilities, and configuration options, see the OmniParse Extension Guide.

DDE MCP tool

Kamiwaza also ships a DDE-focused MCP tool, tool-kamiwaza-dde, for apps and agents that need to work with catalog, ingestion, retrieval, VectorDB, or knowledge-graph operations without building their own REST clients.

Area Example tools What they are used for
Catalog search_catalog, get_dataset, add_to_catalog Discover dataset metadata and register datasets in the catalog.
Pipelines ingest_source, ingest_document, create_pipeline_job, get_pipeline_job Submit content and manage ingestion jobs.
Retrieval search_context, retrieve_context, agentic_search Run grounded context search and retrieval from agents.
VectorDB list_collections, search_collections, query_vectors Inspect and query retrieval collections and vector stores.
Knowledge graph add_knowledge, search_knowledge, get_memory Populate and query Graphiti-backed knowledge state.

The exact tool inventory can vary by release. The current line aligns the MCP surface more closely to the REST APIs exposed by the platform. To inspect the exact tool IDs your deployment exposes, send the standard MCP tools/list request after initialize:

{
  "jsonrpc": "2.0",
  "id": "2",
  "method": "tools/list",
  "params": {}
}

Session and auth flow

The DDE MCP tool uses streamable HTTP MCP semantics:

  1. Send initialize to /mcp.
  2. Read the returned MCP-Session-Id and MCP-Protocol-Version headers.
  3. Send both headers on subsequent requests in the same session.

Operational notes:

Security markings and rate limits

DDE document indexing and retrieval enforce security markings and system-high rules:

Default rate limits are controlled by DDE_DOCUMENT_RATE_LIMIT (requests per window) and DDE_DOCUMENT_RATE_WINDOW_SECONDS.

Operational guidance