Krabka
GitHub krabka-io

Go Streams & Arrow

Stream processing in Go featuring Apache Arrow batch acceleration and barrier cuts via krabka-streams-go.

Full Go API Reference
Installation (Go 1.26+): 
$ go get github.com/krabka-io/krabka-streams-go

Packages

Package Purpose
schemaRegistry client, cache, and Avro/Protobuf/JSON Schema serdes
columnarApache Arrow batch processing, topologies, and barrier cuts
columnarschemaAvro and Protobuf Arrow memory bridges

Columnar Processing Example

Topology Definition: 
package main

import (
	"context"
	"github.com/krabka-io/krabka-streams-go/columnar"
	"github.com/apache/arrow/go/v17/arrow"
	"github.com/apache/arrow/go/v17/arrow/array"
	"github.com/apache/arrow/go/v17/arrow/memory"
)

func main() {
	mem := memory.NewGoAllocator()
	codec := columnar.NewBlobCodec(mem)
	topology := columnar.NewTopology(mem)
	
	source, _ := topology.AddSource("source", []string{"transactions"}, codec)
	large, _ := topology.AddOperator("large",
		columnar.Filter(mem, func(batch arrow.Record, row int) bool {
			return batch.Column(1).(*array.Int64).Value(row) > 4
		}), source)
	topology.AddSink("archive", "large-transactions", codec, large)
	
	built, err := topology.Build()
	if err != nil { panic(err) }
	
	_ = built
}

Automatic GoDoc & API Reference

The complete interactive API reference for Go is generated with Bazel and published automatically to GitHub Pages:

Open krabka-streams-go API Reference