Class AvroArrowSchemas

java.lang.Object
io.krabka.streams.columnar.schema.AvroArrowSchemas
public final class AvroArrowSchemas extends Object
Translates Avro schemas into Arrow schemas.

Records become one column per field, nested records become Struct columns, arrays become List, maps become Map with Utf8 keys, and the decimal, date, time, and timestamp logical types become their native Arrow counterparts. Enums become Utf8 symbol columns and fixed becomes FixedSizeBinary, both tagged with krabka.avro.* field metadata so the value conversion is reversible. A union of null with one branch is that branch made nullable; a union with several non-null branches becomes a Struct with one nullable child per branch of which exactly one is set. A recursive record reference falls back to its JSON text in a Utf8 column tagged krabka.json, because an Arrow schema is a finite tree. A non-record top-level schema maps to a single column named value.

Example

var avro = new org.apache.avro.Schema.Parser().parse("""
    {"type": "record", "name": "Order", "fields": [
      {"name": "id", "type": "string"},
      {"name": "total", "type": {"type": "bytes", "logicalType": "decimal",
                                 "precision": 10, "scale": 2}}]}""");
org.apache.arrow.vector.types.pojo.Schema arrow = AvroArrowSchemas.toArrowSchema(avro);
// columns: id (Utf8, not null), total (Decimal(10, 2, 128), not null)

  • Method Summary

    Modifier and Type
    Method
    Description
    static org.apache.arrow.vector.types.pojo.Field
    toArrowField(String name, org.apache.avro.Schema schema)
    Translates one Avro schema into one Arrow field.
    static org.apache.arrow.vector.types.pojo.Schema
    toArrowSchema(org.apache.avro.Schema schema)
    Translates an Avro schema into the Arrow schema its batches use.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • toArrowSchema

      public static org.apache.arrow.vector.types.pojo.Schema toArrowSchema(org.apache.avro.Schema schema)
      Translates an Avro schema into the Arrow schema its batches use.
      Parameters:
      schema - the Avro reader schema
      Returns:
      the Arrow schema; one column per field for a record schema, one value column otherwise
      Throws:
      ColumnarException - if the schema uses an unsupported shape, such as a decimal precision above 76

    • toArrowField

      public static org.apache.arrow.vector.types.pojo.Field toArrowField(String name, org.apache.avro.Schema schema)
      Translates one Avro schema into one Arrow field.
      Parameters:
      name - the field name the column uses
      schema - the Avro schema of the value
      Returns:
      the Arrow field, nullable when the schema is a union with null
      Throws:
      ColumnarException - if the schema uses an unsupported shape