Interface StatefulColumnarProcessor

All Superinterfaces:
ColumnarProcessor
All Known Implementing Classes:
BuiltinOp
public interface StatefulColumnarProcessor extends ColumnarProcessor
A processor whose partition-local state can be snapshotted and restored.

The topology snapshots stateful processors through BuiltColumnarTopology.snapshotPartition(int) and restores them through BuiltColumnarTopology.restorePartition(int, java.util.Map). Group runners do this automatically around rebalances using a ColumnarStateStore, and roll a partition back to its pre-batch snapshot when processing fails.

The snapshot format is the processor's own; it only needs to be stable between the versions of the application that write and read it. Snapshots should be self-validating: restore(byte[]) throws ColumnarException for bytes it does not understand.

Example

final class RunningCount implements StatefulColumnarProcessor {
    private long count;

    public void process(ColumnarContext context, VectorSchemaRoot batch) {
        count += batch.getRowCount();
    }

    public byte[] snapshot() {
        return ByteBuffer.allocate(Long.BYTES).putLong(count).array();
    }

    public void restore(byte[] snapshot) {
        count = ByteBuffer.wrap(snapshot).getLong();
    }
}

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    restore(byte[] snapshot)
    Replaces the processor's state with a previously taken snapshot.
    byte[]
    snapshot()
    Serializes the processor's current state.

    Methods inherited from interface ColumnarProcessor

    process

  • Method Details

    • snapshot

      byte[] snapshot()
      Serializes the processor's current state.
      Returns:
      the state as bytes; never null, empty for "no state yet"

    • restore

      void restore(byte[] snapshot)
      Replaces the processor's state with a previously taken snapshot.
      Parameters:
      snapshot - bytes previously returned by snapshot()
      Throws:
      ColumnarException - if the bytes cannot be restored