Package io.krabka.streams.columnar
package io.krabka.streams.columnar
Apache Arrow batch codecs, topology operators, runners, and state handling.
This package processes Kafka records as Arrow batches instead of one record at a
time. A BatchCodec decodes a fetched partition
batch into one VectorSchemaRoot, a
ColumnarTopology routes whole batches through
operators, and a ColumnarRunner drives fetch,
process, produce, and commit cycles against plain Kafka clients. Stateful operators
keep partition-local state that can be snapshotted into a
ColumnarStateStore.
Example
try (var allocator = new RootAllocator()) {
var codec = new BlobCodec(allocator);
var topology = new ColumnarTopology(allocator);
var source = topology.addSource("source", List.of("transactions"), codec);
topology.addSink("sink", "processed-transactions", codec, source);
try (var built = topology.build()) {
var output = built.runBatch("transactions", records);
}
}
-
ClassDescriptionBuilt-in within-batch aggregate functions.Defines one aggregation, with an optional explicit output type.A Kafka serde for Arrow IPC stream batches.Type-coercing Arrow batch helpers for
RowBridgeandColumnarProcessorimplementations.Converts one partition batch between Kafka records and Arrow vectors.Converts records whose values are Arrow IPC batches.A validated and reusable columnar topology.Creates the built-in Arrow batch operators.Collects batches that a columnar processor forwards.Selects how a group runner handles a failed logical partition batch.The reaction to a failed partition batch.Reports an Arrow codec, topology, or operator failure.Configuration for an inner, event-time, equi-join.Lock-free counters for a columnar runner.One point-in-time view of the counters.An opaque handle to a columnar topology node.Processes one Arrow batch and forwards zero or more output batches.Runs fetch, process, asynchronously produce, and commit cycles.A consumer-group runner that manages rebalances and persistent operator state.Persists operator snapshots for one logical Kafka partition and one epoch.A closeable snapshot-retention lease without checked exceptions.A linear or branching graph whose edges carry Arrow batches.One input record in a partition batch.Defines a column that a with-columns operator adds or replaces.Stores each partition snapshot in one atomically replaced local file.Adds per-record GZIP compression to any batch codec.Converts JSON-compatible Java records to flat Arrow columns.Binds one produced record to its sink topic.One output record from a columnar sink.One immutable Kafka record header.RowBridge<T>Converts typed rows to and from an Arrow payload batch.RowCodec<T>Converts ordinary Kafka value records through a typed row bridge.Tests one row in an Arrow batch.Computes one derived Arrow column value.A processor whose partition-local state can be snapshotted and restored.