Rust Edition 2024 · Release v1.0.8
Broker Engine Overview
The core streaming broker: request pipelines, write-ahead logs, coordinators, and KRaft consensus via krabka-broker.
krabka-broker implements the Apache Kafka wire protocol in safe Rust, with asynchronous I/O and a KRaft metadata quorum. Selected decision-kernel contracts have Creusot proofs, and Stateright models check failure sequences within stated bounds. The compatibility matrix lists supported APIs and implementation status.
Architecture & Engineering Guides
In-depth technical guides maintained alongside the broker codebase:
- Broker Configuration Reference: Server properties, listener parameters, and storage thresholds.
- Verification & Invariants: Formal mathematical proofs with Creusot and model checking with Stateright.
- Kafka KIP Compatibility Matrix: Kafka Improvement Proposals implementation status and behavioral conformance.
- Docgen Contract: Compiler-enforced doc generation contracts and JSON schema synchronizers.
- Release Process: Build verification steps, SLSA Level 3 provenance attestations, and release signing.
Running a Broker
Docker Run (Single-Node KRaft)
# 1. Initialize KRaft metadata on a fresh storage volume
docker run --rm \
-v krabka-data:/var/lib/krabka \
--entrypoint /usr/bin/krabka-format \
ghcr.io/krabka-io/krabka-broker:latest \
--log-dir=/var/lib/krabka \
--standalone \
--node-id=1 \
--controller-listener=localhost:9093
# 2. Run the broker daemon
docker run -d \
--name krabka \
-p 9092:9092 \
-v krabka-data:/var/lib/krabka \
ghcr.io/krabka-io/krabka-broker:latest \
--listen-addr=0.0.0.0:9092 \
--advertised-listener=localhost:9092 \
--log-dir=/var/lib/krabka \
--process-roles=controller,brokerBroker Crates & Subsystems
| Crate | Purpose & Subsystem |
|---|---|
| krabka-broker | Cluster-capable Kafka-compatible broker: request handlers, coordinators, partition replication, quotas, and the krabka-broker daemon binary |
| krabka-raft | Metadata KRaft quorum (KIP-595 KraftController) |
| krabka-kraft-core | Deterministic, sans-IO KRaft consensus core (KIP-595/996) |
| krabka-log | Byte-compatible reader and writer for Apache Kafka's on-disk log format |
| krabka-logfmt | Structured-JSON tracing log formatter shared across krabka services |
| krabka-remote-storage | KIP-405 tiered-storage SPI (RemoteStorageManager and RemoteLogMetadataManager) with reference implementations; ships the krabka-worm-verify binary |
| krabka-remote-storage-topic | Topic-backed RemoteLogMetadataManager for tiered storage |
| krabka-object-store | Unified object-store construction (S3-compatible, GCS, local filesystem) shared by tiered storage and the observability blockstore |
| krabka-verified | Formally verified pure kernels (Creusot) shared by consensus, log, and security |
| krabka-throttle | Shared token-bucket throttling primitives |
| krabka-authz | Shared Kafka-ACL authorization evaluator for the broker and gateway |
| krabka-audit | Audit event model, OCSF serialization, and write pipeline; ships the krabka-audit verify binary |
| krabka-format | Formats the metadata log and data directories: Kafka's meta.properties, bootstrap records, and the singleton VotersRecord (krabka-format binary) |
| krabka-restore | Offline point-in-time restore: rebuilds a bootable log directory from a KIP-405 tiered-storage archive (krabka-restore binary) |
| krabka-backup | Captures the inputs a point-in-time restore needs (RLMM snapshot, controller checkpoint, committed group offsets) and restores offsets afterwards |
| krabka-records-legacy | Kafka v0/v1 MessageSet codec and v2 RecordBatch bridge |
| krabka-telemetry | OTLP distributed-tracing pipeline and profiling support |
| krabka-barrier | CLI that administers barrier groups: define a group, trigger a cut, list cuts, verify a cut |
| krabka-guard | CLI that administers topic write freezes and break-glass proposals (two-person rule) |
Version & Compatibility
| Component | Supported Version | Notes |
|---|---|---|
| Rust Toolchain | Rust 1.99.0 (Edition 2024) | Pinned stable toolchain; Creusot proofs and cargo +nightly fmt use a nightly |
| Kafka Wire Protocol | Kafka 4.3.1 (KIP-595 KRaft) | Byte-exact request/response schema serialization |
| On-disk formats | Readable across 1.x | Every 1.x broker reads what an earlier 1.x broker wrote; new formats wait for a finalized feature level. Reformat 0.x data directories. See persisted formats. |
| OS Architecture | Linux x86_64, aarch64 | Container image is built on a Wolfi (glibc) base |
krabka-log, krabka-verified and krabka-macros 1.0.0 are on crates.io. The other broker crates, the broker included, are not published; pin this repository by git revision to use them.
Full version matrices and SLSA Level 3 build provenance records are tracked on the Versions page.