Persisted formats

This document is the authoritative list of every format that a krabka broker writes to disk or to an object store, and the compatibility contract that covers them from 1.0.0 on.

The contract covers krabka-broker and the persisted crates of krabka-protocol: krabka-metadata, the record, metadata-envelope and checkpoint codecs of krabka-protocol, and every type in krabka-ids, krabka-voters and krabka-security that a persisted MetadataRecord contains. The other krabka repositories keep their greenfield rule.

Design Goals

• A 1.x broker reads all 1.x data. Any 1.x broker reads every on-disk artifact that any earlier 1.x broker wrote. An operator never reformats a node or restores from a backup to move from 1.x to 1.y.
• A rolling upgrade works. A cluster moves from 1.x to 1.y one node at a time. Mixed-version nodes exchange the krabka-private records of the controller-forwarding RPCs during the roll, so those records are under the same contract as the disk.
• A downgrade works until the operator finalizes. A new broker writes the old format until the operator finalizes the feature level that introduces the new format. Until then, the operator can roll the cluster back to the earlier build.
• The Kafka semantics are the model. The gating follows Kafka’s metadata.version model from KIP-584 and KIP-778, so an operator who knows kafka-features knows the procedure.

Architecture Overview

The contract

1. Read compatibility. A 1.y broker reads every format in the tables below as any 1.x broker with x less than or equal to y wrote it. This covers partition logs and their sidecars, metadata log segments and snapshots, quorum-state, meta.properties, the bootstrap files, the internal-topic record formats, the krabka-private wincode records inside NoOpRecord tags, and the object-store formats: tiered segments, WORM manifests, diskless WAL objects, and backup captures.
2. Gated writes. A change to a persisted format ships behind a metadata.version level or another feature level. A broker that supports the new format keeps writing the old format until the operator finalizes that level, as Kafka’s KIP-584 and KIP-778 semantics require.
3. Downgrade. Before finalization, the operator can downgrade to any earlier 1.x build, because nothing on disk uses the new format yet. After finalization, a downgrade follows Kafka’s rules for the feature in question. For metadata.version, krabka implements the KIP-1155 snapshot path.
4. No promise for 0.x data. Data that a broker before 1.0.0 wrote gets no promise. An operator reformats a 0.x data directory with krabka-format and restores the topic data.

What the contract does not cover

• The Kafka wire protocol. The Kafka compatibility rules in CLAUDE.md are unchanged. A request or a response follows Kafka’s version negotiation, not this document.
• The Rust API. The contract is about bytes. It does not promise semver stability for a Rust type, a function or a crate interface.
• Caches. remote-log-index-cache/ is emptied at startup. It is not durable state.
• Configuration and command-line flags. They follow the normal changelog process.

Key Design Decisions

Rules for a format change

These rules apply to every format in the tables below. CLAUDE.md gives the same rules as instructions to a contributor.

• Wincode values are positional. wincode writes an enum variant as its index and a struct as its fields in order, with no field names. So the order of the MetadataRecord variants, and the field layout of every type a variant contains, is part of the on-disk format. Add a new variant at the end only. Do not reorder, remove or insert a variant, and do not change the fields of a persisted wincode type. To change the shape of a record, add a new variant at the end and keep the old variant readable.
• NoOpRecord private tags are permanent. The tags 1001, 1003, 1004, 1005 and 1006 each carry one krabka-private record. Tag 1002 is a gap in the numbering. Never reuse it. A new private record takes a new tag above 1006.
• A new or changed format carries a version marker. Its reader decodes every earlier 1.x version of that format.
• A new writer behavior is gated. The broker writes the new format only after the operator finalizes the feature level that introduces it.
• A golden-bytes fixture pins each format. A test decodes bytes that an earlier release wrote and compares the result with the expected value.

Gating

A metadata.version level is the gate when a krabka format change happens at the same level as a Kafka change. The metadata.version table mirrors Kafka’s MetadataVersion enum exactly, because a JVM client throws on a level that its enum does not know. So krabka cannot add a metadata.version level of its own.

A krabka-only format change is gated on krabka.version, the feature that krabka owns (krabka_metadata::krabka_version in krabka-protocol). Levels 0 and 1 both mean the 1.0.0 formats: Kafka advertises a feature only when its highest supported level is above 0, so a 1.0.0 node advertises krabka.version at [0, 1], and level 1 can never carry a new format. The first krabka-only format change takes level 2. A fresh cluster carries no krabka.version record: krabka-format bootstraps it at level 0, which means the same formats as level 1, unless --feature krabka.version=N overrides it. Once every node is krabka, an operator finalizes level 1 with kafka-features upgrade --feature krabka.version=1. The level stays at 0 until then because a cluster can mix in Kafka nodes, which support only level 0: a bootstrapped level 1 crashes a JVM standby controller and keeps a JVM broker from joining. The controller refuses a level that a registered node does not support and accepts a safe or unsafe downgrade, as Kafka’s FeatureControlManager does for a feature other than metadata.version and kraft.version. kafka-features upgrade --release-version does not move it, because that walks only Kafka’s production features.

Partition log directory

Each partition directory is <log_dir>/<topic>-<partition>/.

Kafka names a replica move directory <topic>-<partition>.<uuid>-future. format-divergences.md records the difference. The broker writes no recovery-point, replication-offset or cleaner-offset checkpoint, and no partition.metadata.

A sealed .log file is always exactly as long as the batches in it, and so is an active one unless preallocate=true. With preallocate=true, Kafka sets a new segment’s length to segment.bytes and trims it when the segment closes, so after a crash its file can end in zeros. krabka reserves the blocks with fallocate(FALLOC_FL_KEEP_SIZE) without changing the length, and writes the active segment through O_DIRECT, which writes whole blocks: the active segment’s file can run up to one block (at most 4 KiB on the filesystems the broker runs on) of zeros past its last batch. Sealing the segment, turning preallocate off, and a clean shutdown cut the zeros. From 1.0.4, several of a segment’s direct writes can run at the same time and finish in any order, so a crash can also leave zeros between batches that were not yet synced, as page-cache writeback can. In both cases the tail recovery of every 1.x release cuts the file at the first bytes that do not decode as a batch, as Kafka cuts its own preallocated tail. recovery_cuts_the_padding_a_crash_leaves in crates/log/src/log/direct_writes.rs holds that reader to it.

Log directory root

Kafka writes bootstrap.checkpoint instead. The bootstrap checkpoint at offset zero of the metadata log, described below, replaces bootstrap.records.bin for a controller of a dynamic quorum.

Metadata log directory

The metadata partition directory is <metadata_log_dir>/__cluster_metadata-0/.

FeatureLevelRecord persists the finalized metadata.version and the other feature levels. A metadata.version downgrade writes a snapshot at the lower level and discards the incompatible log prefix, as KIP-1155 describes.

Krabka-private metadata records

KIP-631 has no schema for some krabka records. Each one rides as the only tagged field of a Kafka NoOpRecord (apiKey 20), and the field body is a wincode MetadataRecord. Kafka’s tools read the record as a NoOpRecord and keep the tag as an unknown tagged field. crates/metadata/src/kraft_translate.rs in krabka-protocol defines the tags.

The reader checks that each tag carries the variant assigned to it, and refuses a tag of 1001 or above that the table does not assign. A golden-bytes test in krabka-protocol pins the wincode encoding of every MetadataRecord variant and of every enum its fields reach.

bootstrap.records.bin holds wincode MetadataRecord values too, so every variant is on disk, not only the five above.

The broker also reads the topic config key krabka.elr, which a 0.x broker wrote, and removes it from the topic config. No 1.x broker writes it.

Internal topics

The group coordinator writes a __consumer_offsets group record only where Kafka 4.3.1’s GroupMetadataManager writes it. So a reader of the topic must accept two shapes that a write on every transition did not make:

• A group can exist without a group metadata record. A new consumer or share group starts at Kafka’s group epoch 1, and the first ConsumerGroupMetadataValue (type 3) or ShareGroupMetadataValue (type 11) comes only with the first epoch bump. A member that joins without a change to the subscribed topic names writes its member record and its current assignment (types 5 and 8 for a consumer group) and no type 3. Replay creates the group from the first child record, as Kafka’s getOrMaybeCreatePersistedConsumerGroup does, at epoch 1.
• A target assignment can be empty. ConsumerGroupTargetAssignmentMemberValue (type 7), ShareGroupTargetAssignmentMemberValue (type 13) and StreamsGroupTargetAssignmentMemberValue (type 21) can hold an empty assignment. That is a member with no target, which is not the same as a tombstone: the tombstone removes the member’s target, and the empty value keeps it.

Other local state

Object store

Rolling-upgrade RPCs

These records are not on disk, but nodes of two 1.x versions exchange them during a rolling upgrade. The contract covers them.

crates/raft/src/wire.rs defines the three codecs. Their versions are negotiated through krabka.version, as Kafka’s inter-broker protocol versions follow metadata.version: the controller’s ApiVersions answer stays Kafka’s table, byte for byte. A sender sends the version that krabka_metadata::private_rpc_version gives for the level finalized in its metadata image, and v0 with no image or no finalized level. Because the controller finalizes only a level that every registered node supports, every peer serves that version. A receiver serves 0 up to the version at its own highest supported level and answers anything else with UNSUPPORTED_VERSION. Levels 0 and 1 give v0 for all three, so a new RPC version arrives with a new level.

Known gaps

The items below are true of the code at 1.0.0.

Fixture coverage

Every krabka-owned format in the tables above has a golden-bytes test that encodes a value, compares it with fixed bytes and decodes the bytes back. The Kafka-owned formats rely on the Kafka differential suites instead. The replay-fence key __krabka_diskless_replay_fence in __diskless_wal_index is a fixed literal with an empty value and has no version.

Unknown tagged fields in group records

Kafka 4.3.1 builds each group record that it writes from its in-memory group, not from the record it replayed. So it drops the unknown tagged fields at the top level of a replayed record when it writes that record again, and krabka drops them too. KIP-1331’s streams tags 2 and 3, which Kafka 4.3.1 does not know, follow the same rule: krabka writes them only while group.streams.topology.description.plugin.class is set.

Kafka keeps two nested structs of a replayed record as the generated objects, with their unknown tagged fields: ConsumerGroupMemberMetadataValue.ClassicMemberMetadata, with each of its ClassicProtocol entries, and StreamsGroupMemberMetadataValue.Endpoint. A member record that Kafka writes from a copy of the member, such as the record of a static member’s replacement, carries those fields again. And because the generated equals compares them, a classic rejoin or a streams heartbeat that sends the same values as the replayed record still writes the member record again, and for the streams heartbeat also bumps the group epoch. krabka decodes these structs into its own types and drops their unknown tagged fields, so it writes the member record without them, and it writes no record and bumps no epoch for such a heartbeat. Kafka 4.3.1 defines no tagged field in these structs, so only a record from a later Kafka version can carry one.

Integration

• CLAUDE.md gives the rules above as instructions to a contributor.
• Deploy gives the operator procedure for a rolling upgrade and for the finalization of a feature level.
• Releasing says how a release that changes a persisted format records the change.
• format-divergences.md lists where the files that krabka-format writes differ from the files that kafka-storage format writes.

Kafka / KIP Compliance

• KIP-584 and KIP-778 define finalized feature levels and the metadata.version upgrade. The gating rule follows them.
• KIP-631 defines the metadata record frame, and KIP-630 defines the snapshot format.
• KIP-1155 defines the metadata.version downgrade that the controller implements.
• KIP-482 tagged fields let a Kafka tool read a krabka NoOpRecord and keep the private tag as an unknown tagged field.
• KIP-405 defines the tiered-segment layout in the object store.

Testing

• crates/metadata/src/kraft_translate.rs in krabka-protocol pins the private tag list and the bytes of each private record, and crates/metadata/src/wincode_contract.rs pins the wincode encoding of every MetadataRecord variant and nested enum, so a test fails when a tag or a layout changes.
• Each krabka-owned format has a golden-bytes test next to its codec, and a table of refused versions, including the layout a 0.x broker wrote.
• Verification lists the model checks and proofs, including the quorum-state load decision.