Krabka
GitHub krabka-io
← Back to Features Overview

Krabka Connect & Postgres CDC

Stream real-time PostgreSQL database mutations directly into Krabka topics with guaranteed ordering, snapshotting, and transaction boundary preservation without running a heavy external JVM Kafka Connect cluster.

Connector SPI Contract (krabka-connect)

The connector engine defines the core Source, Sink, and Converter traits, with derive macros in krabka-connect-derive for strongly-typed configuration validation.

Source Trait

Polls external systems, producing typed change envelopes with precise transaction metadata and LSN offsets.

Sink Trait

Consumes batches from Kafka topics, mapping them to downstream databases or data lakes with idempotent writes.

Worker Runtime

Drives tasks via krabka-connect-worker, checkpointing progress and resuming without event loss.

PostgreSQL Logical Decoding Engine (krabka-connect-postgres)

The PostgreSQL connector interfaces directly with PostgreSQL logical replication streams using the standard pgoutput decoding plugin:

  • Lock-Free Consistent Snapshot: Performs initial table export without taking table locks that disrupt production queries.
  • Transaction Boundary Preservation: Propagates transaction commit timestamps, commit LSNs, and transaction IDs in Kafka record headers.
  • Automatic Schema Evolution: Maps PostgreSQL table DDL changes directly to Confluent Schema Registry schemas.

PostgreSQL CDC Connector Config

# krabka-connect.toml - PostgreSQL CDC Source Connector
[[connectors]]
name = "postgres-orders-cdc"
type = "io.krabka.connect.postgres.PostgresSourceConnector"
tasks_max = 4

[connectors.config]
database_hostname = "db-prod.internal"
database_port = 5432
database_user = "krabka_cdc"
database_dbname = "ecommerce"
plugin_name = "pgoutput"
publication_name = "krabka_orders_pub"
slot_name = "krabka_orders_slot"
topic_prefix = "cdc.postgres"
tombstones_on_delete = true

Active-Active Replicator (krabka-replicator)

A native, high-throughput replication service wire-compatible with MirrorMaker 2. It synchronizes topic partitions across distinct geographic clusters, handles consumer offset translation, and prevents circular record replication for seamless multi-region disaster recovery.

← Tiered Storage Polyglot Streams →