Interface ColumnarProcessor

All Known Subinterfaces:

StatefulColumnarProcessor

All Known Implementing Classes:

BuiltinOp

Functional Interface:

This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.

@FunctionalInterface
public interface ColumnarProcessor

Processes one Arrow batch and forwards zero or more output batches.

Register a processor with ColumnarTopology.addOperator(String, java.util.function.Supplier, ColumnarNode). The supplier is invoked once per logical partition, so instance fields hold partition-local state; implement StatefulColumnarProcessor when that state must survive rebalances and restarts.

Buffer ownership

• The input batch belongs to the framework; never close it.
• Forwarding the input unchanged is allowed; the framework keeps it alive.
• A batch you create and forward transfers to the framework.
• A batch you create and do not forward is yours to close.
• If process throws, the framework closes the input and everything already forwarded, then rethrows.

Example

ColumnarProcessor dropEmpty = (context, batch) -> {
    if (batch.getRowCount() > 0) {
        context.forward(batch); // forwarding the input is allowed
    }
    // forwarding nothing drops the batch; the framework closes it
};
topology.addOperator("drop-empty", () -> dropEmpty, source);

Method Summary

Modifier and Type	Method	Description
void	process(ColumnarContext context, org.apache.arrow.vector.VectorSchemaRoot batch)	Processes one input batch.

Method Details

process

void process(ColumnarContext context,
 org.apache.arrow.vector.VectorSchemaRoot batch)

Processes one input batch.

Parameters:

context - the collector output batches are forwarded to

batch - the input batch, owned by the framework