Class BlobCodec

java.lang.Object
io.krabka.streams.columnar.BlobCodec
All Implemented Interfaces:
BatchCodec
public final class BlobCodec extends Object implements BatchCodec
Converts records whose values are Arrow IPC batches.

Use this codec when producers already write Arrow IPC streams as record values, so one Kafka record holds many rows. Decoding reads each record value as an IPC stream, attaches that record's key, timestamp, partition, offset, and headers to every row through the reserved metadata columns, and concatenates the results; all records in a batch must share one payload schema.

Encoding drops the metadata columns and serializes the payload back into IPC records: it packs the largest run of consecutive rows that fits under maxRecordBytes and shares one key, timestamp, and header list, and applies that envelope to the produced record. A single row larger than the cap throws ColumnarException instead of producing a record the broker would reject.

Example

var codec = new BlobCodec(allocator); // 900 KiB cap per record

try (var batch = codec.decode("transactions", records)) {
    var amounts = (BigIntVector) batch.getVector("amount");
    var offsets = (BigIntVector) batch.getVector(BlobCodec.OFFSET_COLUMN);
}

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    DEFAULT_MAX_RECORD_BYTES
    The default maxRecordBytes: 900 KiB, safely under a 1 MiB broker limit.
    static final String
    HEADERS_COLUMN
    The reserved column holding each row's encoded Kafka headers, __headers.
    static final String
    KEY_COLUMN
    The reserved column holding each row's record key, __key.
    static final String
    OFFSET_COLUMN
    The reserved column holding each row's source offset, __offset.
    static final String
    PARTITION_COLUMN
    The reserved column holding each row's source partition, __partition.
    static final String
    TIMESTAMP_COLUMN
    The reserved column holding each row's record timestamp, __timestamp.

  • Constructor Summary

    Constructors
    Constructor
    Description
    BlobCodec(org.apache.arrow.memory.BufferAllocator allocator)
    Creates a codec with the 921600-byte record cap.
    BlobCodec(org.apache.arrow.memory.BufferAllocator allocator, int maxRecordBytes)
    Creates a codec with an explicit record size cap.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.arrow.vector.VectorSchemaRoot
    decode(List<ConsumedRecord> records)
    Decodes Arrow IPC record values into one batch with metadata columns.
    List<ProduceRecord>
    encode(org.apache.arrow.vector.VectorSchemaRoot batch)
    Encodes a batch's payload into size-capped Arrow IPC records.
    static String
    payloadColumn(String name)
    Returns the in-batch name used when a payload column collides with metadata.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface BatchCodec

    decode, encode

  • Field Details

    • DEFAULT_MAX_RECORD_BYTES

      public static final int DEFAULT_MAX_RECORD_BYTES
      The default maxRecordBytes: 900 KiB, safely under a 1 MiB broker limit.
      See Also:

    • KEY_COLUMN

      public static final String KEY_COLUMN
      The reserved column holding each row's record key, __key.
      See Also:

    • TIMESTAMP_COLUMN

      public static final String TIMESTAMP_COLUMN
      The reserved column holding each row's record timestamp, __timestamp.
      See Also:

    • PARTITION_COLUMN

      public static final String PARTITION_COLUMN
      The reserved column holding each row's source partition, __partition.
      See Also:

    • OFFSET_COLUMN

      public static final String OFFSET_COLUMN
      The reserved column holding each row's source offset, __offset.
      See Also:

    • HEADERS_COLUMN

      public static final String HEADERS_COLUMN
      The reserved column holding each row's encoded Kafka headers, __headers.
      See Also:

  • Constructor Details

    • BlobCodec

      public BlobCodec(org.apache.arrow.memory.BufferAllocator allocator)
      Creates a codec with the 921600-byte record cap.
      Parameters:
      allocator - the allocator that owns decoded batches

    • BlobCodec

      public BlobCodec(org.apache.arrow.memory.BufferAllocator allocator, int maxRecordBytes)
      Creates a codec with an explicit record size cap.

      Raise the cap only together with the broker's max.message.bytes; leave room for headers and framing.

      Parameters:
      allocator - the allocator that owns decoded batches
      maxRecordBytes - the largest encoded record value the codec may produce
      Throws:
      IllegalArgumentException - if maxRecordBytes is not positive

  • Method Details

    • payloadColumn

      public static String payloadColumn(String name)
      Returns the in-batch name used when a payload column collides with metadata.

      A payload column named like a reserved column, for example __offset, is escaped inside processing batches; sinks restore the original name before encoding. Use this method to address such a column inside operators.

      Parameters:
      name - the original payload column name
      Returns:
      the escaped in-batch name, or name itself when it does not collide

    • decode

      public org.apache.arrow.vector.VectorSchemaRoot decode(List<ConsumedRecord> records)
      Decodes Arrow IPC record values into one batch with metadata columns.
      Specified by:
      decode in interface BatchCodec
      Parameters:
      records - the non-empty records of one topic partition batch
      Returns:
      the concatenated batch; the caller must close it
      Throws:
      ColumnarException - if the record list is empty, a record is not a readable IPC stream, or the records' payload schemas differ

    • encode

      public List<ProduceRecord> encode(org.apache.arrow.vector.VectorSchemaRoot batch)
      Encodes a batch's payload into size-capped Arrow IPC records.

      Each produced record's key, timestamp, and headers come from the reserved metadata columns of the rows it carries; a missing or null __timestamp becomes 0 and a missing or null __key becomes a null key.

      Specified by:
      encode in interface BatchCodec
      Parameters:
      batch - the batch to encode; the codec reads it and leaves it open
      Returns:
      the encoded records in row order
      Throws:
      ColumnarException - if one row alone exceeds maxRecordBytes