Partition rebalancer

Install and start

Start a broker first with the quickstart. Build the service from the pinned source revision, then bind it to localhost for local use. The data directory must be writable.

git clone https://github.com/krabka-io/krabka-rebalancer.git
cd krabka-rebalancer
git checkout 9df7a58ec6f0e78b582e7d69c60b208a9c73644e
cargo build --release -p krabka-rebalancer --bin krabka-rebalancer

./target/release/krabka-rebalancer \
  --bootstrap-servers 127.0.0.1:9092 \
  --listen-addr 127.0.0.1:9300 \
  --data-dir ./rebalancer-data

For Kubernetes, install the chart from the same checkout. fullnameOverride makes the Service name match the operator's default endpoint for cluster demo. The chart uses one replica, a Recreate update strategy, and a persistent volume for local history. Its default image tag comes from Chart.yaml; use image.tag or image.digest for your chosen service build.

# Run from the checked-out krabka-rebalancer repository.
# Replace bootstrapServers with brokers reachable from the pod.
helm upgrade --install demo ./charts/krabka-rebalancer \
  --namespace kafka --create-namespace \
  --set-string fullnameOverride=demo-rebalancer \
  --set-string bootstrapServers=demo-broker-headless.kafka.svc:9092

kubectl -n kafka port-forward service/demo-rebalancer 9300:9300

Create and review a proposal

The service exposes Connect RPC at /krabka.rebalancer.v1.Rebalancer/<Method>. All methods use POST. Check /readyz, read the cluster snapshot, then compute a proposal. Creation stores the plan without changing broker assignments.

curl -fsS http://127.0.0.1:9300/readyz

curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/GetState \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d '{}'

curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/CreateProposal \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d '{"goals":["RackAware","ReplicaDistribution","LeaderDistribution"]}'

Review every movement's topic, partition, old and new replica lists, and leader change. A newLeader value of -1 means Kafka elects the replacement; the proposal does not enforce a particular leader. Use the returned proposal ID for later calls.

# Copy the id returned by CreateProposal after reviewing its movements.
PROPOSAL_ID='replace-with-proposal-id'

curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/DryRunProposal \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d "{\"id\":\"$PROPOSAL_ID\"}"

curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/GetProposal \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d "{\"id\":\"$PROPOSAL_ID\"}"

DryRunProposal returns the stored summary. Its estimatedBytesMoved is currently zero, so it cannot estimate transfer size. A proposal with no movements needs no execution; ordinary full/add execution rejects an empty plan.

Execute, monitor, and cancel

Execute only after checking the plan. The service permits one execution at a time. The response marks the proposal as executing; it does not mean reassignment is complete. Poll GetProposal for completion and inspect failureReason on failure. The throttle is bytes per second per broker direction.

# This request changes broker assignments. Use the reviewed proposal id.
curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/ExecuteProposal \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d "{\"id\":\"$PROPOSAL_ID\",\"throttleBytesPerSec\":\"50000000\"}"

# Poll GetProposal until Completed, Failed, or Cancelled.
curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/GetProposal \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d "{\"id\":\"$PROPOSAL_ID\"}"

To stop an active execution, call CancelExecution. Cancellation can return while the proposal is still executing if its drain timeout expires. Keep polling for Cancelled, Failed, or Completed.

curl -fsS http://127.0.0.1:9300/krabka.rebalancer.v1.Rebalancer/CancelExecution \
  -H 'Content-Type: application/json' \
  -H 'Connect-Protocol-Version: 1' \
  -d "{\"id\":\"$PROPOSAL_ID\"}"

The executor applies the replication throttle, submits reassignments, waits for completion, then attempts to clear the throttle. Check logs after a failure: a throttle cleanup failure is logged even when the proposal reaches a terminal state.

Proposal modes and broker removal

Add and remove modes require unique broker IDs already present in the cluster snapshot. They reject in-flight reassignments, offline partitions, and under-replicated partitions. Removal also requires enough surviving brokers to preserve every partition's replication factor and enough movement budget to evacuate all selected brokers.

Removal is disabled until the service has a nonempty evacuation token or token file. Send Authorization: Bearer <token> when creating, executing, or cancelling a removal proposal. The token file takes precedence and is read on each request, so projected Secret rotation takes effect without a restart. The Helm chart accepts brokerEvacuationAuth.existingSecret and brokerEvacuationAuth.key.

The removal planner selects replacement brokers by replica count. It does not run the rack or capacity goals. Check that the surviving brokers have enough capacity and that the proposed replicas meet your rack placement requirements before approval.

Before executing a removal plan, the service checks the current assignments again. If they changed, create a new proposal and review it. A completed evacuation request with no movements confirms that the selected brokers hold no replicas. Review the resulting cluster state before removing broker workloads. Keep the service reachable only by callers authorized to change cluster assignments; the evacuation credential protects the removal mode.

Goals and broker capacities

An empty goals list selects all 16 registered goals. Hard goals run before soft goals. If the movement cap prevents a hard goal from being satisfied, optimization fails. Soft goals use the remaining movement budget.

Set --broker-capacity-file capacity.yaml to load capacity limits at startup. Missing broker entries or fields mean no limit for that broker or resource. Without a capacity file, the five capacity goals make no changes. Replace the example limits with the measured capacities of your brokers.

version: 1
brokers:
  1:
    max_replicas: 4096
    disk_bytes: 1099511627776
    network_in_bytes_per_sec: 125000000
    network_out_bytes_per_sec: 125000000
    cpu_cores: 8.0
  2:
    max_replicas: 4096
    disk_bytes: 1099511627776
    network_in_bytes_per_sec: 125000000
    network_out_bytes_per_sec: 125000000
    cpu_cores: 8.0

With Helm, use brokerCapacities for the map under brokers; the chart writes the versioned capacity file. brokerCapacityFile overrides that map and requires you to provide the file. Usage goals need broker metrics. The scraper discovers each broker host from metadata and uses port 9404, or you can set static targets such as --metrics-scrape-targets 1:broker-1:9404,2:broker-2:9404.

Configure the service

Flags take precedence over their environment variables. Time values without a -secs suffix accept units, such as 60s and 100ms. Resource counts must be positive where the parser requires them.

Use KafkaRebalance with the operator

Install the operator and its CRDs, and deploy a rebalancer reachable from the operator. The KafkaRebalance field explorer lists its specification and status fields. The cluster label is required. The default endpoint is http://<cluster>-rebalancer.<namespace>.svc.cluster.local:9300; use spec.endpoint to override it.

apiVersion: krabka.io/v1alpha1
kind: KafkaRebalance
metadata:
  name: demo-balance
  namespace: kafka
  labels:
    krabka.io/cluster: demo
spec:
  mode: full
  goals: [RackAware, ReplicaDistribution, LeaderDistribution]
  throttleBytesPerSec: 50000000

The operator computes a proposal and reports ProposalReady. Check status.optimizationResult and use status.sessionId to inspect the full proposal through GetProposal. Approve it with the annotation below. The operator enters Rebalancing, then reports Ready or NotReady.

kubectl -n kafka get kafkarebalance demo-balance -o yaml

# After checking status.optimizationResult and the proposal:
kubectl -n kafka annotate kafkarebalance demo-balance \
  krabka.io/rebalance=approve --overwrite

kubectl -n kafka get kafkarebalance demo-balance -w

# Request a new proposal, or cancel an execution that is running:
# kubectl -n kafka annotate kafkarebalance demo-balance krabka.io/rebalance=refresh --overwrite
# kubectl -n kafka annotate kafkarebalance demo-balance krabka.io/rebalance=stop --overwrite

For spec.mode: addBrokers or removeBrokers, provide spec.brokers. Removal uses the same-namespace Secret <cluster>-rebalancer-auth, key token, unless spec.authorizationSecretRef supplies another name and key. The service and operator must use the same credential. Keep spec.goals absent or empty for removal.

API reference

Use the versioned protobuf schema to generate clients. JSON requests use camelCase field names; protobuf 64-bit integer values can use decimal strings.

Readiness, recovery, and troubleshooting

• GET /healthz checks the running HTTP service. GET /readyz returns 503 until a cluster snapshot exists and the state topic has loaded. GET /metrics exposes OpenMetrics under the krabka_rebalancer prefix.
• Proposal history lives in proposals.json and anomaly history in anomalies.json under the data directory. Preserve this directory across restarts. The internal compacted Kafka topic stores the in-flight execution record.
• If state loading exceeds its timeout, the loader keeps retrying and readiness stays false until loading succeeds. Startup recovery can stop waiting at this deadline; check logs and the proposal state before starting another operation.
• Unavailable means a snapshot or state load is still missing. InvalidArgument includes unknown goal names and incompatible mode inputs. FailedPrecondition includes an unsatisfied goal, unstable cluster state, insufficient surviving brokers, stale removal assignments, an empty plan, or another active execution.
• PermissionDenied during broker removal means evacuation is disabled or the credential does not match. Check the configured token file or Secret without writing its value to logs.

Implementation references: RPC handlers, optimizer safety checks, runtime validation, Helm values, and Rust API documentation.