Class ProtobufArrowSchemas

java.lang.Object
io.krabka.streams.columnar.schema.ProtobufArrowSchemas
public final class ProtobufArrowSchemas extends Object
Translates Protobuf message descriptors into Arrow schemas.

Scalar fields map to their Arrow counterparts, with uint32 and fixed32 widened to signed 64-bit integers and uint64 and fixed64 kept exact as unsigned 64-bit columns. Nested messages become nullable Struct columns, repeated fields become List, and map fields become Map. Enums become Utf8 symbol columns tagged krabka.proto.enum. Fields with presence — message fields, optional scalars, and oneof members — are nullable; proto3 implicit-presence scalars are not, so an unset scalar reads as its default value.

Well-known types get special treatment: google.protobuf.Timestamp becomes a UTC microsecond timestamp, the wrapper types unwrap to nullable scalars tagged krabka.proto.wrapper, and the dynamic google.protobuf.Struct, Value, and ListValue — which recurse and therefore cannot form a finite Arrow tree — are stored as their canonical Protobuf JSON text in a Utf8 column tagged krabka.json and krabka.proto.message. The same JSON fallback applies to any recursive message reference.

Example

org.apache.arrow.vector.types.pojo.Schema arrow =
    ProtobufArrowSchemas.toArrowSchema(Order.getDescriptor());
// e.g. columns: id (Utf8), total_cents (Int(64, true)), placed_at
// (Timestamp(MICROSECOND, "UTC"), nullable)

  • Method Summary

    Modifier and Type
    Method
    Description
    static org.apache.arrow.vector.types.pojo.Field
    toArrowField(com.google.protobuf.Descriptors.FieldDescriptor field)
    Translates one message field into one Arrow field.
    static org.apache.arrow.vector.types.pojo.Schema
    toArrowSchema(com.google.protobuf.Descriptors.Descriptor descriptor)
    Translates a message descriptor into the Arrow schema its batches use.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • toArrowSchema

      public static org.apache.arrow.vector.types.pojo.Schema toArrowSchema(com.google.protobuf.Descriptors.Descriptor descriptor)
      Translates a message descriptor into the Arrow schema its batches use.
      Parameters:
      descriptor - the descriptor of the message type
      Returns:
      the Arrow schema with one column per message field
      Throws:
      ColumnarException - if a field uses an unsupported shape

    • toArrowField

      public static org.apache.arrow.vector.types.pojo.Field toArrowField(com.google.protobuf.Descriptors.FieldDescriptor field)
      Translates one message field into one Arrow field.
      Parameters:
      field - the descriptor of the message field
      Returns:
      the Arrow field, nullable when the message field tracks presence
      Throws:
      ColumnarException - if the field uses an unsupported shape