Skip to main content
krabka
GitHub

Features and architecture

The broker is written in safe Rust. Separate repositories provide clients, stream processing, connectors, and Kubernetes operations. Selected broker decisions have Creusot proofs and Stateright models.

Broker characteristics

Runtime, protocol, verification, and tooling details.

Feature Implementation
Runtime Rust binary with Tokio async I/O
Memory safety unsafe_code forbidden by lint under Cargo and Bazel
Formal verification Creusot-proved decision-kernel contracts in krabka-verified
Model checking Stateright models checked within stated bounds on every pull request
Metadata consensus KRaft quorum with KIP-595 wire codecs
Kafka wire protocol Apache Kafka 4.3 codecs generated from upstream schemas
Admin tooling Apache Kafka kafka-*.sh CLI tools and krabka-cli
Connect and CDC Separate krabka-connect PostgreSQL CDC worker and replicator
License Apache License 2.0

Components

Architecture, crates, and configuration for each component.

krabka-broker Details

Native KRaft consensus quorum

The Rust KRaft quorum stores cluster metadata in krabka-raft and krabka-kraft-core. It implements KIP-595 wire codecs, separate broker and controller roles, and dynamic quorum reconfiguration without ZooKeeper.

  • Byte-exact KRaft metadata records and in-memory image
  • Dynamic voter addition and decommission without cluster restart
  • Election and quorum decisions proved with Creusot; consensus model-checked with Stateright
krabka-broker Details

Tiered storage

krabka-remote-storage and krabka-object-store upload sealed partition segments to Amazon S3, Google Cloud Storage, or MinIO. The __remote_log_metadata topic stores the segment catalog.

  • Retention of sealed segments in object storage
  • Local write-ahead log with async cloud offloading
  • Retention and remote-read decisions are Creusot-proved kernels
krabka-connect Details

krabka Connect and Postgres CDC

The separate krabka-connect worker streams PostgreSQL WAL changes into Kafka topics with checkpointed, at-least-once delivery. Changes to each primary key stay in order; the connector does not take an initial snapshot.

  • PostgreSQL logical replication worker over pgoutput
  • Distributed replicator (MirrorMaker 2 compatible)
  • Schema-aware record transformations and checkpoints
krabka-streams-* Details

Polyglot stream processing and Arrow

Stream processing libraries for Rust, Go, and Java. The libraries provide state stores, Apache Arrow record-batch operators, schema serdes, and barrier coordination; support varies by language.

  • Native Rust streams (Tokio async pipeline and state stores)
  • Go streams with Arrow record batches and barrier cuts
  • Java client with schema registry and Arrow columnar serdes
krabka-operator Details

Kubernetes native operator

The operator reconciles krabka.io/v1alpha1 custom resources for Kafka clusters, node pools, topics, users, connectors, schema registries, and partition rebalancing.

  • CRDs for Kafka, KafkaNodePool, KafkaTopic, KafkaUser, and KafkaConnector
  • StatefulSet rolling updates with readiness checks
  • Integrated partition reassignment and rebalancing engine
krabka-verified Details

Correctness and verification

The broker forbids unsafe Rust. Creusot proves selected decision-kernel contracts, and Stateright checks crash and retry sequences within model bounds. CI reruns the proofs and models on every pull request.

  • unsafe_code forbidden under Cargo and Bazel
  • Creusot proof sessions committed beside the code
  • Stateright models with pinned unique-state counts

Repositories

Broker, clients, streams, and operations.

GitHub organization
Repository Description
Rust krabka-broker Kafka-compatible broker with KRaft and tiered storage.
Rust krabka-protocol Kafka wire codecs, KRaft records, and SASL/TLS.
Rust krabka-client-rs Kafka producer, consumer, and admin APIs.
Rust krabka-streams-rs Stream processing, windowing, and state stores.
Go krabka-streams-go Arrow batch processing and barrier cuts.
Java krabka-streams-java Kafka Streams APIs, Arrow batches, and schema serdes.
Rust krabka-schema-registry Confluent-compatible schemas stored in Kafka.
Rust krabka-operator Kubernetes cluster and resource management.
Rust krabka-connect PostgreSQL CDC and cross-cluster replication.
Rust krabka-rebalancer Partition assignment and cluster balancing.
Rust krabka-cli Cluster, topic, ACL, and consumer-group administration.

Built on krabka

Repository Description
krabka-o11y Metrics, traces, profiles, and logs.
krabka-o11y-demo Grafana dashboards and an example orders pipeline.
gres Postgres-compatible compute backed by krabka and object storage.