Polyglot streams and Apache Arrow
Stream processing libraries for Rust, Go, and Java. The Go and Java libraries process Apache Arrow record batches. The Rust library provides a Tokio runtime, state stores, and KIP-1071 streams-group membership.
Streaming runtimes for multiple languages
Each language has its own stream processing library:
krabka-streams-rs
Async stream builder on Tokio. In-memory and disk-backed state stores with schema registry serdes.
krabka-streams-go
Go stream pipelines with Apache Arrow record-batch operators and barrier cut snapshots.
krabka-streams-java
Kafka Streams 4.x API with Arrow vector batch serdes and columnar operators.
Apache Arrow columnar processing
Decoded record batches use Apache Arrow columnar formats with 5 reserved metadata columns (__key, __timestamp, __partition, __offset, and __headers). Operators filter and transform batches before a sink encodes them back into Kafka records.
Go streams with Apache Arrow columnar batching
package main
import (
"context"
"github.com/krabka-io/krabka-streams-go/columnar"
"github.com/apache/arrow/go/v17/arrow"
"github.com/apache/arrow/go/v17/arrow/array"
"github.com/apache/arrow/go/v17/arrow/memory"
)
func main() {
mem := memory.NewGoAllocator()
codec := columnar.NewBlobCodec(mem)
topology := columnar.NewTopology(mem)
source, _ := topology.AddSource("source", []string{"transactions"}, codec)
large, _ := topology.AddOperator("large",
columnar.Filter(mem, func(batch arrow.Record, row int) bool {
return batch.Column(1).(*array.Int64).Value(row) > 4
}), source)
topology.AddSink("archive", "large-transactions", codec, large)
built, err := topology.Build()
if err != nil { panic(err) }
_ = built
}Barrier cuts and KIP-1071 streams coordination
krabka implements the KIP-1071 streams group coordination protocol. The broker periodically publishes epoch markers across partitions to generate a barrier cut on __barrier_state for snapshots of multi-stream application state.