Application Gateway

krabka-gateway accepts application requests over Connect-RPC and HTTP, then uses native Kafka clients to reach the brokers. Use it for browser applications, functions, webhook receivers, and queue workers.

Install and run locally

The source build requires Rust 1.99.0 or newer. The workspace pins its sibling dependencies; build from the repository root. The gateway crate is not published to crates.io at this revision.

git clone https://github.com/krabka-io/krabka-gateway.git
cd krabka-gateway
git checkout 49870fd8e5f708274ff69a3e64a092844e2f8809
cargo build --release -p krabka-gateway --bin krabka-gateway

Start a broker first and create the topics your application will use. Then run the gateway. Both the broker bootstrap list and the advertised gateway address are required.

KRABKA_BOOTSTRAP_SERVERS=127.0.0.1:9092 \
KRABKA_GATEWAY_LISTEN_ADDR=127.0.0.1:9500 \
KRABKA_GATEWAY_ADVERTISED_ADDR=127.0.0.1:9500 \
./target/release/krabka-gateway

# In another terminal
curl --fail http://127.0.0.1:9500/healthz
curl --fail http://127.0.0.1:9500/readyz

The listener above uses local plaintext HTTP. /healthz checks liveness. /readyz returns HTTP 503 until the deduplication state is ready, and HTTP 200 when the replica can accept traffic.

The container image is ghcr.io/krabka-io/krabka-gateway. Main branch builds use commit SHA tags; releases use version tags. Pin a tag or digest from the container registry. The image runs as user 65532 and contains no shell.

For Kubernetes, use the operator's KafkaGrpcGateway reference. The operator documentation covers installation and reconciliation.

Send a record over HTTP

POST /v1/produce/{topic} writes the request body to the named topic. An optional Idempotency-Key enables deduplication. The caller identity comes from listener authentication; with no authenticated identity it is ANONYMOUS.

# Create the events topic before sending records.
curl --fail-with-body http://127.0.0.1:9500/v1/produce/events \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: example-delivery-1' \
  --data-binary '{"message":"hello krabka"}'

A successful response contains partition, offset, and deduplicated. Retry an uncertain delivery with the same idempotency key. Use a new key for each distinct delivery.

Connect-RPC contract and SDKs

The protobuf service contract defines krabka.gateway.v1.Gateway. The gateway mounts Connect-RPC handlers for JSON and protobuf clients.

Records accept raw bytes or a structured JSON body, optional keys, ordered headers, partitions, timestamps, and idempotency keys. Send acknowledgment modes are ACKS_ALL, ACKS_LEADER, and ACKS_NONE. Check per-record errors even when the RPC succeeds.

With SubscribeStart.auto_commit=false, send a SubscribeAck only after durable processing. Its offset is the delivered record's offset. The gateway commits the contiguous acknowledged frontier plus one; an acknowledgment above a gap cannot skip unfinished records. Automatic mode ignores explicit acknowledgments.

Queue sessions live in one gateway process. Leave session_id empty on the first acquisition, then reuse the returned ID on the same replica. The message limit is fixed when the session starts. This revision allows at most 1,024 sessions per process, 500 messages per acquisition, and a 30-second wait. lock_duration_ms must be 30000; sessions expire after one minute of inactivity. Broker acquisition expiry controls redelivery.

Application SDKs are available in Rust, C++, Go, Java, and TypeScript. The repository guide lists toolchains and shared conformance checks. Use native clients when your application needs Kafka administration or a direct broker connection.

Schemas and subscription filters

Set --schema-registry-url to a Confluent-compatible registry to encode structured records as Avro, JSON Schema, or Protobuf. A record's schema selector can specify a subject and registered ID. An empty subject uses <topic>-value; a zero ID resolves the latest schema. Without a registry, raw bytes pass through.

A subscription's filter is a SQL boolean expression over decoded fields, such as status = 'PAID' AND total > 100. Leave it empty to deliver all authorized records. Arrow filtering requires the gateway's arrow feature, which is enabled in the default source build.

Webhook ingress and CloudEvents

Load --webhooks-config with a TOML file containing [[endpoints]]. Each endpoint supplies a name, target topic, service principal, body limit, and optional signature settings. Requests go to POST /v1/webhooks/{name}. Unconfigured names return HTTP 404.

Generic HMAC-SHA256 verification supports a signature header, hex or base64 encoding, and an optional prefix. key_source and idempotency_source accept a header name or a JSONPath expression. Named endpoints default to a 1MiB body limit. Use the endpoint configuration reference for all fields.

For Standard Webhooks, use signature_mode="standard_webhooks" with the complete signing token. The verifier authenticates the delivery ID, timestamp, and original body, then uses the signed webhook-id for deduplication. Preserve these bytes and headers through your proxy. The GitLab guide covers secrets, replay tolerance, replica settings, and consumer recovery.

The gateway accepts CloudEvents binary and structured HTTP formats. Binary events store attributes in ce_* Kafka headers. Structured events retain their JSON envelope and content type. CloudEvents batch requests return HTTP 415. Follow the CloudEvents guide for both ingress formats and the local round trip.

Outbound webhook delivery

Load --outbound-webhooks-config with source topics, a target URL, a retry policy, and an allowlist of target schemes and hosts. An empty allowlist denies all targets. Create the source and dead-letter topics before starting delivery.

[[allowed_targets]]
scheme = "https"
host = "hooks.example.com"

[[subscriptions]]
name = "events-delivery"
source_topics = ["events"]
target_url = "https://hooks.example.com/events"
dead_letter_topic = "events-delivery-dlq"
max_attempts = 5
base_backoff = "500ms"
max_backoff = "30s"
request_timeout = "10s"
content_mode = "cloud_events_structured"

This example requires incoming records to carry valid CloudEvent attributes. Use content_mode="envelope" for the gateway JSON envelope, or cloud_events_binary for binary CloudEvents. decode_to_json=true uses the configured registry codec to decode values; it has no effect with the raw codec.

Each subscription uses its own consumer group unless you set group_id. Delivery retries use exponential backoff. With no dead-letter topic, the gateway logs and drops exhausted records after max_attempts. Configure a dead-letter topic when failed deliveries must be retained. Optional signing_secret adds an X-Krabka-Signature HMAC header; static headers can supply target authentication. Protect configuration files that contain secrets.

Outbound filters accept json:<JSONPath>. Header filters are rejected. See the outbound configuration reference for defaults and validation.

Deduplication across replicas

The gateway stores the payload and deduplication claim in one transaction. Replicas divide claim partitions through an ownership group, publish membership, and forward requests to the current owner. Claims must replay before a new owner accepts traffic. The default deduplication horizon is one hour.

Give every replica a unique client ID and a peer-reachable advertised address. Set an explicit shared ownership group when client IDs differ, and share the dedup topic, partition count, membership topic, and transaction prefix across that gateway deployment. Separate deployments need their own internal topic and transaction namespace.

The default internal-topic replication factor is three, with fallback enabled for smaller clusters. For a deployment that requires three replicas, set --internal-topic-allow-replication-fallback=false and verify topic replication and minimum ISR before accepting traffic. The gateway does not repair existing topic durability settings. The GitLab deployment guide provides explicit three-broker settings.

Deduplication ends when its retained claim expires. Consumers must make their own durable effects safe to repeat. An HTTP timeout can occur after a transaction commits, so retries must preserve the delivery ID.

Security

The defaults use plaintext connections and --authz=off. Configure listener TLS with both --tls-cert and --tls-key. For mTLS, supply --tls-client-ca and --tls-client-auth=required. The same gateway certificate identifies the forwarder to peers; configure peer trust roots and use a certificate suitable for both roles.

Use --authz=simple to enforce ACL checks. Listener authentication provides the caller's principal; named webhooks use the endpoint's configured service principal, which defaults to webhook:<name>. Grant only the required topic and group permissions. Super-user names bypass ACL checks.

--bearer=unsecured accepts unsigned JWTs for development and tests. This configuration surface does not provide signed JWKS validation. Do not use unsecured bearer validation as production authentication.

At this revision, broker TLS is enabled only when both --broker-tls-cert and --broker-tls-key are set, with --broker-tls-server-name. A broker CA file alone leaves the connection plaintext. Check the broker security builder before configuring your deployment.

Operations and troubleshooting

• Route application traffic only to replicas passing /readyz. A live but unready replica may still be replaying ownership state.
• Scrape /metrics for request counts and latency, dedup hits, forwarding outcomes, transaction results, owned partitions, and webhook delivery failures. Metrics use the krabka_gateway_ prefix.
• For readiness or forwarding failures, check broker connectivity, internal topic settings, ownership-group configuration, peer addresses, and TLS trust.
• For HTTP 404 on a webhook, check its configured name. For HTTP 401, check signature bytes, token, headers, and clock freshness. HTTP 413 means the body exceeds the endpoint limit.
• Webhook and outbound configuration files are read at startup. Roll replicas after a secret or configuration change. Listener certificates use the configured reload interval.
• Shutdown marks the replica unready and drains active connections for up to eight seconds. Test response loss, owner replacement, broker failover, and rolling shutdown with your application's traffic before deployment.

Configuration reference

CLI flags override environment variables. Supply units for dimensions, such as 500ms, 1h, 2MiB, and 1%; bare numbers are rejected for these values. Dedup retention and internal-topic time settings must resolve to exact whole milliseconds. Pass explicit values for boolean runtime options, such as --internal-topic-allow-replication-fallback=false.

The CLI definitions and runtime defaults specify the configuration at this revision. Run krabka-gateway --help for your installed binary.

Guides and examples