System Design · Architectural Model

Ecosystem Architecture & Layering

Repository boundaries and dependencies across the krabka-io organization.

The broker, clients, operators, and connectors live in separate repositories. Runtime services communicate through public APIs and the Kafka wire protocol. Some build-time dependencies cross these boundaries; for example, the CLI uses the broker's log-formatting crate.

Each tier builds on the tiers below it; arrows point from a layer to what it depends on.

Layering Principles

Client and protocol dependencies: The krabka-operator uses client and protocol crates to send requests to the broker.
Independent Wire Layer: krabka-protocol defines exact byte-level request/response codecs generated from upstream JSON specifications with zero broker or client dependencies.
CLI subcommands: krabka-cli compiles in administration commands and the Gres tenant registry. Other subcommands run as standalone executables found on PATH.

Repository Hierarchy & Subsystems

Repository	Architectural Tier	Subsystems & Purpose
krabka-protocol	Tier 0: Core Wire Layer	Wire codecs, KRaft metadata records, SASL/TLS, compression (snappy/lz4/zstd)
krabka-client-rs	Tier 1: Foundation Clients & Broker	Tokio async Kafka client (Producer, Consumer, Admin)
krabka-broker	Tier 1: Foundation Clients & Broker	Broker engine, KRaft consensus (krabka-raft), on-disk log (krabka-log), tiered storage
krabka-schema-registry	Tier 1: Foundation Clients & Broker	Confluent-compatible schema registry backed by the compacted _schemas topic
krabka-streams-rs	Tier 2: Stream Processing Engines	Rust stream runtime with schema registry serdes and disk-backed state stores
krabka-streams-go	Tier 2: Stream Processing Engines	Go stream processing with Apache Arrow batch memory & barrier cuts
krabka-streams-java	Tier 2: Stream Processing Engines	Java stream processing with Kafka Streams 4.x and columnar operators
krabka-connect	Tier 3: Data Integration & Control Plane	Connector framework, PostgreSQL CDC over pgoutput, worker runtime & replicator
krabka-operator	Tier 3: Data Integration & Control Plane	Kubernetes operator reconciling krabka.io/v1alpha1 custom resources
krabka-rebalancer	Tier 3: Data Integration & Control Plane	Cruise-Control-style partition assignment and cluster balancing daemon
krabka-cli	Tier 4: Tooling	Admin and cluster log directory formatting CLI tool

Explore Architectural Deep Dives

Runtime Architecture →

Operational topology, internal broker components, and control plane reconcile loops.

Correctness & Verification →

Safe Rust, Creusot-proved kernels, and Stateright models across the layered repositories.