Class ArrowIpcSerde

java.lang.Object
io.krabka.streams.columnar.ArrowIpcSerde
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.kafka.common.serialization.Serde<org.apache.arrow.vector.VectorSchemaRoot>
public final class ArrowIpcSerde extends Object implements org.apache.kafka.common.serialization.Serde<org.apache.arrow.vector.VectorSchemaRoot>
A Kafka serde for Arrow IPC stream batches. The caller must close decoded roots.

Each record value is one complete Arrow IPC stream containing exactly one record batch. This is the value format BlobCodec reads and writes; use the serde directly with plain Kafka producers and consumers that exchange Arrow batches without a columnar topology. The serde only handles record values: configuring the deserializer for keys throws IllegalArgumentException.

Example

var serde = new ArrowIpcSerde(allocator);
var producer = new KafkaProducer<>(config, new ByteArraySerializer(), serde.serializer());
producer.send(new ProducerRecord<>("arrow-batches", batch));

// consumer side: every deserialized root must be closed
try (VectorSchemaRoot received = serde.deserializer().deserialize("arrow-batches", bytes)) {
    process(received);
}

  • Constructor Summary

    Constructors
    Constructor
    Description
    ArrowIpcSerde(org.apache.arrow.memory.BufferAllocator allocator)
    Creates a serde whose deserialized batches are owned by an allocator.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.kafka.common.serialization.Deserializer<org.apache.arrow.vector.VectorSchemaRoot>
    deserializer()
    Returns the deserializer half of this serde.
    org.apache.kafka.common.serialization.Serializer<org.apache.arrow.vector.VectorSchemaRoot>
    serializer()
    Returns the serializer half of this serde.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.kafka.common.serialization.Serde

    close, configure

  • Constructor Details

    • ArrowIpcSerde

      public ArrowIpcSerde(org.apache.arrow.memory.BufferAllocator allocator)
      Creates a serde whose deserialized batches are owned by an allocator.
      Parameters:
      allocator - the allocator that owns every deserialized batch's buffers

  • Method Details

    • serializer

      public org.apache.kafka.common.serialization.Serializer<org.apache.arrow.vector.VectorSchemaRoot> serializer()
      Returns the serializer half of this serde.

      The serializer writes one Arrow IPC stream per batch and throws SerializationException when the batch cannot be written. A null batch serializes to null.

      Specified by:
      serializer in interface org.apache.kafka.common.serialization.Serde<org.apache.arrow.vector.VectorSchemaRoot>
      Returns:
      the reusable serializer

    • deserializer

      public org.apache.kafka.common.serialization.Deserializer<org.apache.arrow.vector.VectorSchemaRoot> deserializer()
      Returns the deserializer half of this serde.

      Every non-null result is a freshly allocated root that the caller must close. Malformed streams and streams without a record batch throw SerializationException. Null input deserializes to null.

      Specified by:
      deserializer in interface org.apache.kafka.common.serialization.Serde<org.apache.arrow.vector.VectorSchemaRoot>
      Returns:
      the reusable deserializer