Serdes & Data Formats

Pick the encoding your topic carries, then hand it to the topology.

A serde turns a Rust value into the bytes on a topic and back. The schema serdes resolve schema IDs against a Confluent-compatible registry and write the Confluent wire format, so a Java consumer reads what a Rust producer wrote. The columnar serdes are self-describing Arrow IPC and need no registry.

Serde	Rust type	Cargo feature	Use it for
StringSerde / I64Serde / BytesSerde	String / i64 / Bytes	built in	Primitive keys and values.
SchemaSerde<T, JsonSerde<T>>	any serde type that derives schemars::JsonSchema	built in	Confluent JSON Schema.
SchemaSerde<T, ProtobufSerde<T>>	a prost Message	built in	Confluent Protobuf, resolved through prost-reflect.
SchemaSerde<T, AvroSerde<T>>	apache_avro::AvroSchema	built in	Confluent Avro.
PolarsIpcSerde	polars::DataFrame	polars	Columnar values as Arrow IPC.
ArrowIpcSerde	arrow::RecordBatch	arrow	Interchange with arrow-rs.
ColumnarSerde<T>	columnar::Columnar	columnar	Zero-copy native columnar values.

Enable the ones you need

Cargo.toml:

[dependencies]
krabka-streams-rs = { version = "0.4", features = ["polars", "arrow"] }

The primitive and schema serdes are always available. The three columnar serdes each sit behind a Cargo feature, so a build that does not touch Arrow does not compile Arrow.

Two processing models

Model	Build it with	Test it with	Notes
Row model	Processor API and the high-level DSL	TopologyTestDriver	Processes one record at a time. This is the model that matches Kafka Streams.
Columnar model	ColumnarTopology with Polars DataFrame edges	ColumnarTestDriver	Vectorized aggregation over batches instead of single records.

Both test drivers run a topology with no broker, so a unit test needs no cluster.

The registry service that the schema serdes resolve against is documented on Schema Registry & Serdes. The client and the topology API are on Rust Streams & Client.