Architectural Superpowers
Every layer of Krabka is written in memory-safe Rust to maximize CPU efficiency, minimize latency jitter, and simplify cloud-native operations.
Krabka vs JVM Apache Kafka
A side-by-side comparison of runtime characteristics and operational footprint.
| Feature | Krabka | Apache Kafka (JVM) |
|---|---|---|
| Runtime | Native Rust Binary (tokio async) | JVM (OpenJDK 21) + GC |
| Broker Memory Working Set | 114–622 MiB RSS | 2.5–5.5 GiB Heap + Off-Heap |
| Cold Start to Ready | 1.2 seconds | 8–9 seconds |
| Memory Safety | Safe Rust (unsafe forbidden) | JVM-managed / manual tuning |
| Metadata Consensus | Native KRaft quorum (KIP-595 wire) | KRaft |
| Kafka Wire Protocol | Apache Kafka 4.3 (byte-exact codecs) | Apache Kafka 4.3 |
| Admin Tooling | Standard kafka-*.sh CLI tools | Standard kafka-*.sh CLI tools |
| Connect & CDC | Built-in Postgres CDC worker & replicator | Separate Kafka Connect cluster |
| License | Apache License 2.0 | Apache License 2.0 |
Modular Subsystems Deep Dive
Explore the individual architectures, crates, and implementation specifics of each module.
Native KRaft Consensus Quorum
Metadata lives in a native Rust Raft consensus quorum from day one (krabka-raft & krabka-kraft-core). Implements KIP-595 wire interoperability with split broker/controller roles, dynamic quorum reconfiguration, and zero ZooKeeper dependencies.
- Byte-exact KRaft metadata records & in-memory image
- Dynamic voter addition and decommission without cluster restart
- Fast leader elections with sub-second failover
Tiered Storage with NVMe Caching
Decouple compute from storage (krabka-remote-storage & krabka-object-store). Stream older partition segments automatically into cloud object storage (Amazon S3, Google Cloud Storage, or MinIO) backed by the __remote_log_metadata topic.
- Infinite partition retention at cloud object-storage costs
- Local write-ahead log with async cloud offloading
- Zero impact on real-time produce throughput
Krabka Connect & Postgres CDC
Built-in change data capture (CDC) and connector runtime. Stream PostgreSQL WAL mutations directly into Krabka topics with guaranteed ordering, snapshotting, and transaction boundary preservation.
- PostgreSQL logical replication worker over pgoutput
- High-throughput distributed replicator (MirrorMaker 2 compatible)
- Schema-aware record transformations and checkpoints
Polyglot Stream Processing & Arrow
Purpose-built stream processing engines for Rust, Go, and Java. Native support for Apache Arrow columnar memory enables zero-copy record batch decoding and barrier cut snapshots (KIP-1071 streams group protocol).
- Native Rust Streams (tokio async pipeline & state stores)
- Go Streams with Arrow batch acceleration & barrier cuts
- Java client with Schema Registry and Arrow columnar serdes
Kubernetes Native Operator
Manage your streaming fleet declaratively. Reconciles custom resources in group krabka.io/v1alpha1 for Kafka clusters, node pools, topics, users, listeners, connectors, schema registry, and automated cluster partition rebalancing.
- CRDs for Kafka, KafkaNodePool, KafkaTopic, KafkaUser, KafkaConnector
- Automated pod rolling updates with zero downtime
- Integrated partition reassignment and rebalancing engine
Full-Signal Observability
Telemetry is a first-class citizen (krabka-telemetry). Emits OpenTelemetry traces, PromQL metrics, continuous CPU/memory profiles, and structured JSON/logfmt logs with zero external sidecars required.
- Native PromQL metrics endpoints on /metrics
- Distributed request tracing with OpenTelemetry & W3C trace-context
- Pre-built Grafana 11 dashboards in krabka-o11y-demo