Skip to main content
krabka
GitHub
Back to features overview

krabka Connect and Postgres CDC

Stream real-time PostgreSQL database mutations directly into krabka topics with checkpointed, at-least-once delivery and no separate Kafka Connect cluster. Each change is keyed by the row's primary key, so changes to one row stay in order.

Connector SPI contract (krabka-connect)

The connector engine defines the core Source, Sink, and Converter traits, with derive macros in krabka-connect-derive for strongly typed configuration validation.

Source trait

Polls external systems, producing typed change records and a checkpointable read position such as a PostgreSQL LSN.

Sink trait

Pushes records into an external system in batches, with an optional transactional write gate for exactly-once delivery.

Worker runtime

Runs one PostgreSQL source connector per process via krabka-connect-worker, checkpointing progress in a compacted Kafka topic and resuming after a restart.

PostgreSQL logical decoding engine (krabka-connect-postgres)

The PostgreSQL connector interfaces directly with PostgreSQL logical replication streams using the standard pgoutput decoding plugin:

  • Changes only: There is no initial snapshot. The connector emits the changes that logical decoding makes available from its replication slot.
  • Transaction metadata: Every record value carries the source LSN, transaction ID, and commit timestamp. Deletes are emitted as tombstones.
  • Schema Registry framing: Keys and values are Protobuf, framed with schema IDs from a Confluent-compatible registry. The connector registers one shared change-record schema at startup, and the acceptance suite adds a column mid-stream without interrupting the flow.
Postgres-compatible compute

gres is part of the broader krabka ecosystem and is documented in its own repository.

Visit gres

PostgreSQL CDC worker invocation

# Each krabka-connect-worker process runs one connector. Every flag also has a
# KRABKA_* environment variable; the connection URL is a secret, so pass it there.
export KRABKA_POSTGRES_URL='postgres://cdc_user:[email protected]:5432/inventory'

krabka-connect-worker \
  --connector-id postgres-inventory-cdc \
  --kafka-bootstrap kafka:9092 \
  --schema-registry-url http://schema-registry:8081 \
  --postgres-slot krabka_inventory_slot \
  --postgres-publication inventory_publication \
  --postgres-schema public \
  --postgres-tables orders,line_items,customers

Active-active replicator (krabka-replicator)

A native replication service wire-compatible with MirrorMaker 2. It synchronizes topic partitions across distinct geographic clusters, handles consumer offset translation, and prevents circular record replication for active-passive or active-active multi-region failover.

Tiered Storage Polyglot Streams