Polyglot Streams & Apache Arrow

Purpose-built streaming engines for Rust, Go, and Java. Native integration with Apache Arrow columnar memory enables zero-copy deserialization straight into AI/ML dataframes and analytic pipelines, with barrier cuts and KIP-1071 streams group protocol.

Multi-Language Streaming Runtimes

Instead of wrapping JVM foreign function interfaces or subprocess pipes, Krabka delivers native stream processing libraries for major cloud languages:

krabka-streams-rs

Async stream builder on Tokio. In-memory and disk-backed state stores with schema registry serdes.

krabka-streams-go

High-concurrency Go stream pipelines with native Apache Arrow zero-copy record batch decoding and barrier cut snapshots.

krabka-streams-java

Drop-in compatibility with JVM Kafka Streams 4.x with Arrow vector batch serdes and columnar operators.

Apache Arrow Columnar Processing

Decoded record batches are structured directly into Apache Arrow columnar formats with 5 reserved metadata columns (__key, __timestamp, __partition, __offset, and __headers). This allows analytical filters, vector transformations, and DuckDB/Polars queries to execute with zero-copy SIMD acceleration.

Go Streams with Apache Arrow Columnar Batching

package main

import (
	"context"
	"github.com/krabka-io/krabka-streams-go/columnar"
	"github.com/apache/arrow/go/v17/arrow"
	"github.com/apache/arrow/go/v17/arrow/array"
	"github.com/apache/arrow/go/v17/arrow/memory"
)

func main() {
	mem := memory.NewGoAllocator()
	codec := columnar.NewBlobCodec(mem)
	topology := columnar.NewTopology(mem)
	
	source, _ := topology.AddSource("source", []string{"transactions"}, codec)
	large, _ := topology.AddOperator("large",
		columnar.Filter(mem, func(batch arrow.Record, row int) bool {
			return batch.Column(1).(*array.Int64).Value(row) > 4
		}), source)
	topology.AddSink("archive", "large-transactions", codec, large)
	
	built, err := topology.Build()
	if err != nil { panic(err) }
	
	_ = built
}

Barrier Cuts & KIP-1071 Streams Coordination

Krabka implements the next-generation KIP-1071 streams group coordination protocol. The broker periodically publishes epoch markers across partitions to generate a barrier cut on __barrier_state, enabling atomic snapshotting of multi-stream application state.

← Postgres CDC Kubernetes Operator →