Operations

The operator guide for krabka-broker: what to deploy, what to watch, and what to do when an alert fires. Everything here describes the broker on this branch. Where a document names a flag, a config key or a series, that name is the one the code reads, and the contract check fails the build when the series drift.

Guides

• Deploy: install the image, the binary or the reference Kubernetes manifests. Format a log directory, write the config file, and start, stop and roll a cluster. Wire the /healthz and /readyz probes.
• Capacity: what a broker spends per partition and per byte, how to size disk, network and voters, and how to read the phase families.
• Performance qualification: reproduce the Kafka comparison and the 1,000/10,000-partition operating-envelope run.
• Broker performance results: the measured comparison and highest passing partition tier, with exact provenance.
• Scale a cluster: add capacity, move replicas with a throttle, and drain and unregister a broker, including the operator-managed path.
• Metrics: every exported series with its labels and the JMX name it replaces.
• Ecosystem support: which tool classes work over the Kafka protocol, and which need broker-side JMX or a JVM plugin and therefore do not. Cruise Control is in the second list; the lag monitors that decode __consumer_offsets are in the first.
• Migrate from Apache Kafka: how to move a running Kafka cluster onto krabka with MirrorMaker 2, and how to cut consumers and producers over. Names what the mirror carries, what it does not, and what krabka answers to MirrorMaker 2’s ACL sync.
• Backup and restore: what to copy off each node, how often, and how to check that a copy would restore. The archive holds the records and nothing else, so the RLMM snapshot, the controller metadata checkpoint and the committed group offsets have to be captured before the day they are needed.
• Audit: which admin RPCs write an AdminOperation record to __krabka_audit, what each record names, and what is redacted from it.

Monitoring

• grafana-dashboard.json: a reference Grafana dashboard. Import it and pick the Prometheus datasource it asks for.
• alert-rules.yaml: Prometheus alert rules with the thresholds the metric doc comments recommend. Each rule names its runbook.
• metrics-body.txt: a /metrics body with one label set per family, generated by the metrics_contract test suite. It is what the CI docs job checks the dashboard and the rules against.

Runbooks

One per condition the registry tells operators to alert on, and one for the data-loss event that no alert fires for: restore-from-archive, the rebuild of a cluster out of its tiered-storage archive.

Keeping the contract

aspect check-metrics-contract fails on any series the dashboard or the rules name that the body does not carry. bazel test //crates/broker:metrics_contract_test runs the same check against a live registry and fails when metrics-body.txt is stale. When a family changes, regenerate the body, then update metrics.md, the dashboard and the rules in the same change.