Operator Pilot Acceptance Plans Design

Two workload plans that an operator who did not build krabka follows to install it, migrate onto it from Apache Kafka 4.3.1, cut over, roll back, and recover, with the checks that decide whether the run passed.

This document is the M29 contract for #1398. #1399 executes it. It names no operator and arranges no access. The maintainer actions table lists what a person has to do before a run can start. Each TBD(Mn) marker in this document is a value that action Mn supplies.

Design Goals

• Two access patterns that fail differently. Plan A is a high-throughput event pipeline. It stresses partition count, fan-out, compaction, the remote tier and schema-encoded records. Plan B is a transactional stream processor. It stresses exactly-once visibility, per-key order, the next-generation group protocol and certificate identities. A defect that one plan hides, the other plan should show.
• Only published instructions. The operator follows the documents under docs/operations/ at one pinned revision. A step that needs knowledge that is not in those documents is a defect in the documents, and the run records it.
• Criteria fixed before the run. Every number that decides a verdict is in this document, or in the sign-off record that fills its TBD markers, before the run starts. This is the rule of the sustained-load contract: change a criterion in a reviewed commit, never after a result.
• Existing tools only. The plans use the krabka image and its bundled tools, the krabka-operator chart, the JVM admin tools and MirrorMaker 2 from the Apache Kafka 4.3.1 distribution, and the clients the ecosystem support matrix names. They add no harness, no framework and no script.
• Honest labels. An external pilot and an independent acceptance run produce evidence of different strength. The evidence says which one it is, and an independent run never counts as an external pilot.

Architecture Overview

A run is a sequence of phases. Each phase has entry checks, the runbook sections that the operator follows, and exit checks. A phase that fails its exit checks stops the run. The operator does not skip forward.

A run passes when every phase passes and the intervention log has no entry for P1 or P6. P1 is the installation, and P6 is the recovery drill. The M29 completion criteria need both of them “without maintainer intervention”. An intervention in P2 to P5 does not fail the run, but each one opens a fix, and #1399 reruns the affected phase after the fix is verified.

Every run produces one evidence set:

Key Design Decisions

Choose the workloads from the ecosystem matrix

Both plans use only rows of the What works table: the admin command line, the JVM clients and librdkafka, the protocol-based lag monitors, and MirrorMaker 2 with a schema registry. Neither plan uses a tool from the What does not work table. A pilot that needed Cruise Control or a JMX exporter would measure a known limit, not the migration.

Plan A takes the “consumer-lag monitors” row. Those monitors decode __consumer_offsets themselves, and the matrix calls them the class with the sharpest failure mode. Plan B takes the transaction and group rows that the matrix covers through the JVM clients: Kafka Streams with exactly_once_v2 (KIP-98, KIP-447) and a KIP-848 consumer. The suites jvm_streams_app, jvm_consumer_group_next_gen and mtls_principal_mapping already cover each part in isolation. The pilot covers them together, under an operator.

Plan B does not use share groups (KIP-932). A share group gives no per-key order, and per-key order is the property that Plan B exists to check. The KIP matrix also pins the share-group evidence to Kafka 4.1.0 only.

Use two different installation paths

Plan A installs on Kubernetes with the krabka-operator chart that the qualified manifest names. Plan B installs the container image with a TOML file on hosts that the operator manages, as Deploy describes. The two paths share no installation step, so one plan cannot hide a defect in the other plan’s path. Both paths start from an image digest, never from a mutable tag.

Use two different cutover styles

Plan A uses the procedure of Migrate from Apache Kafka as it is written. MirrorMaker 2 runs with DefaultReplicationPolicy, so mirrored topics carry the prod. prefix. Consumer groups move one at a time, and the producers move last. Per-key order across the cutover point is not a requirement of an event pipeline, so the short period when a consumer reads both the mirrored topic and the new local topic is acceptable.

Plan B cannot accept that period. A Kafka Streams aggregation that reads one key from two topics can count a record twice or in the wrong window. Plan B uses IdentityReplicationPolicy, a shadow run of the Streams application on krabka, and one quiesced switch. All writers stop, the mirror drains, the shadow output is compared with the source output, and then every client moves in one declared window.

Measure loss with canary ledgers from Kafka’s own tools

An operator’s production application is not an instrument. It may not log what it acknowledged, and its owner may not consent to share its data. So each plan adds canary topics that run beside the real workload through every phase. kafka-verifiable-producer writes sequential integer values and prints one producer_send_success JSON line for each acknowledged record. kafka-verifiable-consumer prints the offsets it consumed and committed. Both tools ship in the Apache Kafka 4.3.1 distribution, and the jvm_acceptance_sasl suite already runs kafka-verifiable-producer against krabka. The ledgers give an exact count of acknowledged, mirrored, consumed and re-read records without a new tool. The operator’s own metrics give the throughput and latency criteria.

Tie rollback to persisted-format eligibility

There are two kinds of rollback, and the plans keep them apart.

• Migration rollback moves clients back to the Apache Kafka source. Each plan defines where that stops being possible.
• Version rollback moves krabka from build N to build N-1. #1396 defines its eligibility from Persisted formats. Within 1.x, a roll back to an earlier build works until the operator finalizes a feature level that the earlier build does not support. Rolling upgrade states the same rule.

So both plans forbid kafka-features upgrade during the run, except where a plan step names it. The operator records kafka-features describe at P1 and at the end of each phase, and a change between two records is a finding. If the maintainer ships a fix build during the run, the operator rolls it with Rolling upgrade. That roll stays version-rollback-eligible only while the finalized levels do not change. When #1396 publishes its rollback eligibility table, the sign-off record links the row for the candidate (action M7).

Recover into an isolated target

#1397 destroys source data, but only in its own isolated qualification cluster. A pilot never destroys an operator’s data. Each plan’s drill restores the newest verifying capture from the production archive into a scratch target, as Check a copy and the restore-from-archive runbook describe. The drill then reconciles the scratch cluster against the production cluster at the capture boundary. The drill measures RPO and RTO with the definitions of the disaster-recovery qualification gate, so a pilot result and a #1397 result use the same units. rpo_ms is the age of the capture when the drill declares the loss. rto_ms runs from the declared loss to a cluster that has served every captured record, each topic’s settings and the group positions back to a reader.

A drill reads a live archive. After the capture, the cluster keeps copying new segments to the archive, and remote retention keeps deleting the oldest ones. krabka-restore with --rlmm-snapshot accepts both changes, as Check a copy describes. Each restored partition then starts at the oldest segment the archive holds when the restore reads it. So the drill reads each canary partition’s earliest offset with kafka-get-offsets --time -2 twice: on the scratch cluster, and on production after the restore has finished. Retention only moves production’s earliest offset forward, so the scratch value must not be above it. A restore that stops for one of the reasons that section lists is rerun from a new capture. The first such stop is not a finding. A repeat is.

The archive bounds what a drill can restore. A record reaches the archive only when its segment rolls and the remote copy completes. Compacted topics cannot be tiered, so no archive holds them. The plans set segment.ms on every tiered topic to bound the first limit, and they protect compacted topics another way (see gap issues).

Define a maintainer intervention narrowly and record every one

M29 asks for an install and a recovery drill “without maintainer intervention”. The term needs a definition that a reviewer can check from the evidence.

What counts as a maintainer intervention

A maintainer intervention is any of these, from a person who has built or maintained krabka, or from an agent that acts for that person:

1. A command that the maintainer runs on the pilot’s clusters, hosts or buckets, or a configuration value that the maintainer writes there.
2. Information that changes what the operator does next, and that the published documents at the plan revision do not contain. A private message, a call, a screen share and a public issue comment all count.
3. A build, a patch, a configuration file or an image that is not the declared candidate.
4. A decision about an ambiguous runbook step or about a check result, made for the operator.
5. A change to a success criterion, a deadline or a TBD value after P0.

These are not interventions:

• A maintainer answer that points only to a section that already exists in the documents at the plan revision. The log still records it as a findability finding.
• A maintainer action that the plan itself requires, such as the sign-off in P0 and the evidence verification in P7.
• An escalation that a runbook’s own Escalate section tells the operator to make. The answer to the escalation is an intervention if it meets one of the five conditions above.

How an intervention is recorded

The operator, or the independent runner, keeps the intervention log as a part of the evidence. A maintainer who intervenes tells the operator at once, so the log is complete. Each entry has:

Plan A: High-Throughput Event Pipeline

Workload

The workload is an order and click event pipeline. Many producers append events, several independent consumer groups read every event, and one compacted topic holds the latest profile of each customer. Values are Avro, registered in a schema registry.

The partition count stays inside the 10,000-partition tier that Broker performance results records as passing. That page states that the tier is the highest tested, not a supported limit, and the plan does not claim more.

Clients and versions

Security

• The client listener is SaslSsl with a server certificate (one-way TLS) and SCRAM-SHA-512. The inter-broker listener follows the operator’s own network policy.
• The authorizer is type = "simple", with allow_everyone_if_no_acl_found = false. The admin principal is the only super_users entry.
• Each producer, consumer group, the registry, the lag monitor, MirrorMaker 2 and krabka-backup has its own SCRAM principal and only the ACLs it needs.
• The operator creates each SCRAM principal on krabka as a KafkaUser, which gives it a generated password in a Secret. MirrorMaker 2 does not carry credentials. A client that has to keep its source password gets it set with kafka-configs instead, and the evidence records which principals took which path.

Installation

The operator installs the krabka-operator chart and the broker image by digest from the candidate’s qualified manifest, as Kubernetes operator describes. The Kafka resource sets tieredStorage for the archive bucket, a client listener with tls: true and scram-sha-512, and authorization of type: simple. Each principal is a KafkaUser, and each topic is a KafkaTopic. The operator then runs the krabka-backup capture CronJob from Capture the inputs, hourly, with the backup principal.

Installation exit checks:

1. /readyz answers 200 on every broker, and kafka-metadata-quorum describe --status names one leader and every voter.
2. krabka_broker_under_replicated_partitions and krabka_broker_offline_partitions_count are zero on every broker.
3. A client with a wrong password fails to authenticate. A client with a valid credential and no ACL gets TOPIC_AUTHORIZATION_FAILED. To make the alerts fire, the operator repeats each attempt for longer than its rule’s hold time. KrabkaAuthenticationFailures needs more than 1 failure per second over 5 minutes, held for 5 minutes, so the wrong-password client tries at least 2 times per second for 12 minutes. KrabkaAuthorizationDenials needs a non-zero 5-minute rate held for 10 minutes, so the denied client tries at least once every 30 seconds for 12 minutes. Each alert fires, the operator follows authentication-failures and authorization-denials, and each alert resolves after the attempts stop.
4. The first krabka-backup verify exits 0.
5. kafka-features describe output is recorded.

Migration and cutover checks

MirrorMaker 2 follows Configure MirrorMaker 2 with DefaultReplicationPolicy, sync.group.offsets.enabled = true, offset.lag.max = 100 and offset-syncs.topic.location = target. The operator creates the target topics before MirrorMaker 2 starts. Each target topic has the source’s partition count and the target’s tier and retention settings, and the flow’s config.properties.exclude keeps those settings, as Create the target topics first describes. MirrorMaker 2 uses the client settings and the principal ACLs of Secured clusters. The _schemas topic is mirrored, and the target registry reads it with leader.eligibility=false until the producer cutover, as Cut the schema registry over describes. Both registries stay in READONLY mode until the rollback window closes.

Mirror parity checks, at the end of P2:

Cutover follows Cut the consumers over and then Cut the producers over. Consumers subscribe to the pattern (prod\.)?<topic>, so after the producer cutover they read both the mirrored topic and the local topic <topic> that the producers now write.

Cutover checks, for each consumer group:

Success criteria

The observation window P4 is 7 days on krabka after the last producer moves.

Rollback boundaries

P5 takes place after the producer cutover, so it exercises the third row, on the canary topic only. The operator starts the reverse flow for the local canary topic, moves the canary producer and the canary consumer back to the source, and checks that every value acknowledged on krabka arrives on the source as krabka.<canary topic>. The rollback checks are the same per-partition skipped-record and re-read checks as the cutover, in the other direction. The canary then moves back to krabka, and the reverse flow stops. Version rollback follows Tie rollback to persisted-format eligibility.

Recovery drill

The drill is a full restore rehearsal from the production archive into a scratch cluster on separate storage. It follows Check a copy and, for each step, the restore-from-archive runbook: Confirm, Diagnose and steps 1 to 5 of Fix. Step 6 and step 7 do not apply to a scratch target. The drill uses --cluster-id with the production cluster id and no time bound.

Reconciliation checks:

Runbook sections the operator follows

Plan B: Transactional Stream Processing

Workload

The workload is a payment ledger. Upstream services write payment events in transactions. A Kafka Streams application re-keys them by account, counts and sums them in 1-minute tumbling windows, and writes the results. A downstream service reads the results with read_committed and needs per-account order. The topology has the shape of the jvm_streams_app suite’s application: source, selectKey, repartition, windowed aggregate, sink.

Clients and versions

Security

• The client listener is Ssl with client_auth = "Required" and a client_ca_path. Every client presents a certificate.
• principal_mapping_rules maps the Subject DN to a short name, for example RULE:^CN=([a-z0-9-]+),OU=payments,.*$/$1/L followed by DEFAULT. KIP-371 defines the rule syntax, and the configuration reference states where krabka reads the rules.
• The authorizer is type = "simple". The ACLs name Topic, Group and TransactionalId resources for each principal. The Streams principal has Write and Describe on its TransactionalId prefix and Read on its group.
• The operator installs certificates with the operator’s own PKI. Certificate rotation during P4 is part of the alert-response exercise.

Installation

The operator installs the broker image by digest on three hosts, formats each log directory with krabka-format and a KIP-853 initial voter set, and writes the TOML file. Format the log directory, Configure and Start describe each step. The krabka-backup capture job runs hourly through the operator’s own scheduler with an mTLS client properties file, as Secured clusters describes.

Installation exit checks:

1. The same quorum, health and replication checks as Plan A.
2. A client with no certificate fails the TLS handshake. A client with a certificate from another CA fails the handshake. A valid client whose mapped principal has no ACL gets an authorization error.
3. kafka-acls --list shows every ACL under the mapped short names, not under full DNs.
4. kafka-features describe shows transaction.version and group.version at the levels that the plan’s clients need. If M6 picks group.protocol=streams, it also shows streams.version at level 1 or higher; the operator finalizes it in P1, before the run forbids feature upgrades, because the broker answers the KIP-1071 heartbeats with UNSUPPORTED_VERSION without it. The output is recorded.

Migration and cutover checks

MirrorMaker 2 runs with IdentityReplicationPolicy, offset-syncs.topic.location = target and sync.group.offsets.enabled = false. It mirrors the input topics only. It does not mirror the output topic or the Streams internal topics, and it does not sync the Streams group or the downstream group. The source-side consumer of MirrorMaker 2 reads with isolation.level=read_committed, so the mirror carries no aborted record. Transactional sources and Kafka Streams describes each of these settings, and Secured clusters gives the client settings and ACLs for MirrorMaker 2’s principal.

The cutover has three stages.

1. Shadow. The operator creates the output topic on krabka first, with its 48 partitions and its tier settings, because a Streams sink topic has to exist and broker auto-creation would give it the wrong shape. Then the operator starts the Streams application on krabka with the same application.id and auto.offset.reset=earliest. It reads the mirrored inputs, rebuilds its state, and writes the output topic on krabka. The source application keeps running and stays the system of record.
2. Quiesce. The operator stops the upstream producers on the source and starts the write freeze. The declared freeze is TBD(M2), proposed 15 minutes or less. The mirror drains. Both Streams applications reach zero lag. kafka-transactions list on both clusters shows no ongoing transaction.
3. Switch. The operator compares the outputs, then points the upstream producers and the downstream consumer at krabka, then stops the source application and MirrorMaker 2.

Shadow and quiesce checks:

Switch checks:

Success criteria

The observation window P4 is 7 days on krabka after the switch.

Rollback boundaries

P5 exercises the third row with the canary pipeline only: canary producers, the canary Streams application and the canary consumer. The checks are the shadow and switch checks in the other direction, with the one declared repeat for each window. Version rollback follows Tie rollback to persisted-format eligibility. In Plan B, a change of transaction.version or group.version is also a change of client behavior, so the operator does not change either level during the run.

Recovery drill

The drill is a point-in-time restore, which a bad-write incident needs. The operator picks a timestamp T inside P4 and restores the input topics only, with --topic, into a scratch cluster with --to-timestamp T. The drill follows restore-from-archive: Confirm, Diagnose with the --dry-run plan, and steps 1 to 5 of Fix.

The Streams changelog is compacted, so the archive does not hold it. A restore also does not keep in-flight transaction state, as What comes back states. So the drill creates an empty output topic on the scratch cluster and starts the canary Streams application there with a new application.id. The application rebuilds its state from the restored inputs and writes a fresh output. The drill does not restore the production output topic, so the rebuilt output has no restored records mixed into it.

Reconciliation checks:

Runbook sections the operator follows

Independent Acceptance Runs

An independent acceptance run is the fallback when no external operator is available for a plan by the date in action M9. It runs the same plan, phases, checks and criteria. It is weaker evidence than an external pilot, and the evidence says so.

Who may run one

The runner is a person who meets every condition below. The maintainer records the check in the sign-off record.

• The runner did not author, co-author or approve a change to the paths that the plan exercises in the 6 months before the run. The paths are docs/operations/, crates/backup-cli/, crates/restore/, crates/format/, packaging/k8s/, the krabka-operator repository for Plan A, and the broker handlers for the plan’s features. git log and the pull request review history are the record.
• The runner is not the primary maintainer.
• The runner gets no briefing beyond this document and the published documents at the plan revision.
• The runner is a person. An AI coding agent is not a runner. A runner may use an agent as a tool, and the command log records each use. An agent that has built krabka features cannot supply independence.

Workload

An independent run has no operator workload, so the runner builds the plan’s workload from tools that already exist:

• The source cluster is the pinned apache/kafka:4.3.1 image, at the digest in MODULE.bazel.
• Plan A load comes from kafka-producer-perf-test and the canary tools, at the rates in the workload table. kafka-producer-perf-test sends raw bytes only, so the schema-encoded share of the load comes from kafka-avro-console-producer, which ships in the pinned Confluent Schema Registry image and registers and looks up its schemas in the registry.
• Plan B load comes from the canary tools and transactional-message-copier, which is Kafka’s own exactly-once read-process-write tool, one for each input partition, with --enable-random-aborts. No existing tool runs a Kafka Streams topology at the Plan B rate for 7 days: the application of the jvm_streams_app suite processes five records and stops. So an independent run of Plan B covers the transaction, visibility, order and security criteria, and it does not cover the windowed-output criteria. The evidence lists those criteria as not covered. Action M11 asks for a long-running Streams workload if the maintainer wants them covered.

How the evidence is labeled

• The issue that holds the run is titled acceptance: Plan <A|B>, independent acceptance run <n> (not an external pilot). An external pilot uses pilot: Plan <A|B>, external pilot <n>.
• The first line of every evidence file, report and summary is Independent acceptance run. Not an external pilot.
• A summary, a release note or a roadmap update that counts pilots counts independent runs on a separate line. An independent run never satisfies a criterion that names an external pilot or an operator.

Maintainer Actions

A person does each of these through the project’s normal process. Neither this document nor an agent can do them. Actions marked blocks have to be complete before P0 ends.

Gap Issues

Writing the plans found these gaps and defects in the existing documents. Each one is a small issue in milestone 29. The pull request that added this document also fixed the issues whose Blocks column says so. The validation of the runbooks by independent operators stays with #1399.

Integration

• Deployment. Plan A depends on the krabka-operator chart and image in the qualified manifest, and on the [remote_storage] and [authorization] tables of the broker configuration. Plan B depends on krabka-format, the TOML listener tables and principal_mapping_rules.
• Migration. Both plans depend on MirrorMaker 2 as the mirror_maker2 suite runs it: the stock 4.3.1 distribution against krabka, with OffsetCommit for the translated group positions and IncrementalAlterConfigs for the topic configuration sync.
• Backup and restore. Both drills depend on krabka-backup capture, verify and restore-offsets, and on krabka-restore, as Backup and restore and KFC-3 define them.
• Lifecycle and recovery qualification. The plans take version rollback eligibility from #1396 and the RPO and RTO definitions and declared values from #1397. A pilot repeats neither qualification. It shows that an operator can do the same work from the published documents.
• Sustained load. The alert-response deadlines come from bench/soak/rf3-72h.yaml, so a pilot and the 72-hour soak of #1393 judge failover with the same numbers.

Kafka / KIP Compliance

• MirrorMaker 2 (KIP-382). Both plans run the unmodified 4.3.1 MirrorMaker 2. The offset translation and its replay bound are MirrorMaker 2’s behavior, not krabka’s. The plans measure the re-reads and never accept a skipped record. The ACL sync behavior that Plan A relies on is MirrorMaker 2’s own: it copies topic ACLs and does not copy a WRITE grant.
• Transactions (KIP-98, KIP-447, and KIP-890). Plan B checks read_committed visibility and the absence of hanging transactions with Kafka’s own tools. A transaction never crosses clusters. The quiesce stage makes sure that none is open at the switch, because MirrorMaker 2 carries no transaction state.
• Groups (KIP-848, and KIP-1071 if M6 picks it). Plan B’s downstream consumer uses group.protocol=consumer. Plan A keeps the classic protocol, so the two plans cover both group protocols.
• Security (KIP-371, and KIP-554). Plan A creates SCRAM principals as KafkaUser resources, or with kafka-configs where a client keeps its source password. Plan B maps certificate DNs with ssl.principal.mapping.rules, which krabka reads per listener as principal_mapping_rules.
• Tiered storage (KIP-405). As in Kafka, a compacted topic cannot be tiered. The plans keep compacted topics local and protect them outside the archive.
• Feature levels (KIP-584, KIP-778). Version rollback eligibility follows the finalized levels, as in Kafka.

Testing

The plans are not code, so no suite tests them. Three things keep them honest:

• The suites behind each step. Each step uses a path that an existing suite already runs on every change or in the container lane: mirror_maker2 for the mirror and the offset translation, jvm_streams_app and jvm_consumer_group_next_gen for Plan B’s clients, mtls_principal_mapping for the principal rules, jvm_acceptance_sasl for SCRAM and the canary producer, and crates/restore/tests/dr_roundtrip.rs for the restore path. The KIP matrix links each suite.
• The qualification gates. The ecosystem qualification gates installation, operator-lifecycle, disaster-recovery and registry-migration exercise the candidate that a pilot installs. A pilot does not start on a candidate that has not passed them.
• The rehearsal. Before the first external run, the maintainer runs each plan once against the pinned apache/kafka:4.3.1 source to confirm that every check can be measured with the named tools. That rehearsal is labeled maintainer rehearsal. It is neither a pilot nor an independent acceptance run, and its intervention log is not counted.