Interface BatchCodec

All Known Implementing Classes:
AvroBatchCodec, BlobCodec, GzipBatchCodec, ProtobufBatchCodec, RowCodec
public interface BatchCodec
Converts one partition batch between Kafka records and Arrow vectors.

decode turns a fetched batch into one Arrow batch whose payload columns are followed by the five reserved metadata columns (__key, __timestamp, __partition, __offset, __headers); encode turns an Arrow batch back into records, dropping the metadata columns and applying their values to the produced records. Two implementations ship with the module: BlobCodec for records that already hold Arrow IPC batches and RowCodec for ordinary one-value-per-record topics. GzipBatchCodec wraps any codec with per-record compression.

The caller owns and must close every root returned by decode.

Example

BatchCodec codec = new BlobCodec(allocator);
try (VectorSchemaRoot batch = codec.decode("transactions", records)) {
    List<ProduceRecord> output = codec.encode("processed", batch);
}

  • Method Summary

    Modifier and Type
    Method
    Description
    default org.apache.arrow.vector.VectorSchemaRoot
    decode(String topic, List<ConsumedRecord> records)
    Decodes a fetched partition batch with the source topic available to the codec.
    org.apache.arrow.vector.VectorSchemaRoot
    decode(List<ConsumedRecord> records)
    Decodes a fetched partition batch into one Arrow batch.
    default List<ProduceRecord>
    encode(String topic, org.apache.arrow.vector.VectorSchemaRoot batch)
    Encodes an Arrow batch with the sink topic available to the codec.
    List<ProduceRecord>
    encode(org.apache.arrow.vector.VectorSchemaRoot batch)
    Encodes an Arrow batch into producible records.

  • Method Details

    • decode

      org.apache.arrow.vector.VectorSchemaRoot decode(List<ConsumedRecord> records)
      Decodes a fetched partition batch into one Arrow batch.
      Parameters:
      records - the non-empty records of one topic partition batch
      Returns:
      the decoded batch with metadata columns appended; the caller must close it
      Throws:
      ColumnarException - if a record cannot be decoded

    • decode

      default org.apache.arrow.vector.VectorSchemaRoot decode(String topic, List<ConsumedRecord> records)
      Decodes a fetched partition batch with the source topic available to the codec.

      Topologies call this form so serde-backed codecs can derive registry subjects from the topic. The default implementation ignores the topic and delegates to decode(List).

      Parameters:
      topic - the topic the records were fetched from
      records - the non-empty records of one topic partition batch
      Returns:
      the decoded batch with metadata columns appended; the caller must close it
      Throws:
      ColumnarException - if a record cannot be decoded

    • encode

      List<ProduceRecord> encode(org.apache.arrow.vector.VectorSchemaRoot batch)
      Encodes an Arrow batch into producible records.
      Parameters:
      batch - the batch to encode; the codec reads it and leaves it open
      Returns:
      the encoded records in row order
      Throws:
      ColumnarException - if the batch cannot be encoded

    • encode

      default List<ProduceRecord> encode(String topic, org.apache.arrow.vector.VectorSchemaRoot batch)
      Encodes an Arrow batch with the sink topic available to the codec.

      Topologies call this form so serde-backed codecs can derive registry subjects from the topic. The default implementation ignores the topic and delegates to encode(VectorSchemaRoot).

      Parameters:
      topic - the topic the records will be produced to
      batch - the batch to encode; the codec reads it and leaves it open
      Returns:
      the encoded records in row order
      Throws:
      ColumnarException - if the batch cannot be encoded