Skip to main content
krabka
GitHub

krabka-format and kafka-storage format

This document lists every place where krabka-format differs from kafka-storage format, and why. It is the expected-difference list for a conformance comparison between the two commands: a difference named here is intended, and any other difference is a bug.

The Kafka behaviour below was observed with kafka-storage format on the apache/kafka:4.3.1 image, and read from StorageTool.scala, Formatter.java, and MetaPropertiesEnsemble.java at the 4.3.1 tag. The meta.properties rows were read from MetaProperties.java, MetaPropertiesEnsemble.java, PropertiesUtils.java, and KafkaRaftServer.scala in the Kafka checkout that the system tests pin.

Design Goals

  • An id moves unchanged. A cluster id or a directory id that one tool prints is accepted by the other tool. An operator copies ids between krabka and the Kafka tools, such as kafka-metadata-quorum, so the string form has to be the same.
  • The Kafka tools read the directory. krabka-format writes Kafka’s meta.properties, so a Kafka tool that reads a formatted directory reads a krabka directory too. kafka-metadata-quorum add-controller reads the directory id of the new controller from the meta.properties in its metadata log directory.
  • One run formats the node. A node with several disks is formatted by one command, and each directory gets the same cluster id and its own directory id.
  • A rerun is safe. An init container runs the formatter on every pod start. A run over formatted directories changes nothing, and a run after a failed run finishes the job.
  • Differences are deliberate. Where krabka keeps its own behaviour, this document says why.

Architecture Overview

kafka-storage format reads log.dirs and metadata.log.dir from the server.properties file that --config names. It loads the meta.properties of every directory, checks that the set agrees, and then writes the bootstrap files and meta.properties into each directory that has no meta.properties.

krabka-format has the same structure. It takes the directories from --metadata-log-dir and --log-dir, reads every meta.properties, applies Kafka’s consistency checks, and writes each unformatted directory. --metadata-log-dir is the metadata log directory. Without it, the first --log-dir is the metadata log directory, as Kafka’s metadata.log.dir defaults to the first entry of log.dirs. Only the metadata log directory gets the bootstrap files, and the __cluster_metadata-0 checkpoint of a dynamic KIP-853 format. A data directory gets only meta.properties.

Key Design Decisions

The divergence table

Topic kafka-storage format krabka-format Reason
Cluster id form Writes and prints the id as given. Uuid.randomUuid output is 22 characters of unpadded base64url. Writes and prints Kafka’s 22-character base64url form. Accepts that form, which Uuid.fromString accepts, and the hyphenated form. Matches for every id Kafka generates. Kafka stores any string as cluster.id, for example -t foo. Krabka’s broker holds the cluster id as 16 bytes, so it accepts only a string that decodes to 16 bytes, and normalises the hyphenated form to the base64 form.
Reserved cluster ids Accepted: -t AAAAAAAAAAAAAAAAAAAAAA succeeds. Accepted. Matches. Only Uuid.randomUuid avoids the reserved ids, and so does krabka-format when it generates one.
--cluster-id required Required. Optional. Without it, the id of an already formatted directory in the set is kept, else a new id is generated. A single-node format is one command. kafka-storage random-uuid is not needed first.
Directory ids directory.id in base64url, generated per directory, never reserved or starting with -. The voter directory takes its id from --initial-controllers. The same. --initial-controllers takes either id form. Matches.
Node id node.id from --config, required. meta.properties records it as an int. A negative id is You must specify a valid non-negative node ID. --node-id, required. It takes 0 to 2147483647, and a negative id gets Kafka’s message. Matches the range and the file. The flag follows from --config. A missing --node-id is a clap usage error, exit 2.
--directory-id No such flag. Sets the metadata log directory’s id. An orchestrator that checks the node incarnation before readiness has to know the id before the format runs.
--config Required. Names the server.properties that supplies node.id, process.roles, log.dirs, metadata.log.dir, controller.listener.names, and controller.quorum.voters. Not supported. Krabka’s broker is configured by flags, environment variables, and a TOML file. It does not read server.properties, so a --config would be a second configuration format that only the formatter reads. The values that kafka-storage format takes from the file are flags here: --log-dir, --metadata-log-dir, --node-id, --controller-listener, --controller-listener-name for the first entry of controller.listener.names, and the quorum-mode flags. --help states this.
unstable.feature.versions.enable Read from --config. Off, --release-version 4.4 is Unknown metadata.version '4.4'. Supported metadata.version are: 3.3-IV3, ..., 4.3-IV0 and --feature metadata.version=31 is metadata.version 4.4-IV0 is not yet stable.; on, the default release is latestTesting. --unstable-feature-versions-enable, with the same messages and the same default. Its table of levels ends at 4.5-IV0 (34), as trunk’s does, so with the flag on the default release is 4.5-IV0 and --feature metadata.version=35 is No MetadataVersion with feature level 35. Valid feature levels are from 7 to 34. Follows from --config. The node needs the same setting in its server_properties to support the level it was formatted at.
--release-version and --feature validation --release-version keeps the first two dot-separated segments (4.3.1 is 4.3, 4.0.0 is 4.0, 3.8.x is 3.8) and does not trim. Then, in this order, the release, every feature name, and every feature level are checked. An unknown name is Unsupported feature: <name>. Supported features are: eligible.leader.replicas.version, group.version, kraft.version, share.version, streams.version, transaction.version, without metadata.version. A level a feature does not define is No feature:<name> with feature level <n>. --feature metadata.version=<n> outside the table is No MetadataVersion with feature level <n>. Valid feature levels are from 7 to 31. on 4.3.1, and ... is not yet stable. only for a level in the table past 4.3-IV0. A repeated --feature keeps the last level. The same strings, the same order, and the same lookup key. A repeated --feature is refused as feature <name> specified more than once. The mode conflicts of kraft.version (Cannot set kraft.version to ...) name Kafka’s controller.quorum.voters, which krabka has no equivalent of, so krabka’s texts name --standalone, --initial-controllers, and --no-initial-controllers instead. Matches the release lookup, the names, and the level texts. The kraft.version mode conflicts follow from the flags replacing --config. Refusing a repeated --feature is a krabka rule that Kafka does not have. Kafka raises the No feature and No MetadataVersion errors as an uncaught IllegalArgumentException; krabka prints only the message, with exit 5.
Log directories log.dirs and metadata.log.dir from --config. The set is log.dirs with metadata.log.dir added, each path once. metadata.log.dir defaults to the first entry of log.dirs. --log-dir, repeated or comma-separated, and --metadata-log-dir. The set is the metadata log directory and every --log-dir, each path once. Without --metadata-log-dir, the first --log-dir is the metadata log directory. A metadata log directory that is also a --log-dir is formatted once, as the metadata directory. Matches the set and the default. The flags follow from --config. The broker’s --metadata-log-dir, --log-dir, and --extra-log-dirs name the same directories.
Metadata log directory with an unreadable meta.properties The run does not check it. It prints I/O error trying to read log directory <dir>. Ignoring... and formats the other directories. The broker then refuses to start with Encountered I/O error in metadata log directory <dir>. Cannot continue. The run refuses with the broker’s message, exit 3, and writes nothing. An unreadable data directory is skipped as in Kafka. A node without a readable metadata log directory cannot start, so krabka-format reports it before it writes the other directories.
Directory with foreign files and no meta.properties Formatted. Refused, exit 3. A directory with only the files of an interrupted krabka-format run counts as empty. A mistyped path would otherwise seed a directory full of another program’s data. The exception for an interrupted run keeps a rerun safe.
Already formatted, no --ignore-formatted Log directory <dir> is already formatted. Use --ignore-formatted to ignore this directory and format the others., exit 1. Nothing is written. The same message on stderr, exit 3. Nothing is written. Matches, except for the exit code (see below).
--ignore-formatted over a mixed set Formats the unformatted directories and exits 0. All of the log directories are already formatted. when none is left. The same, and a line on stdout for each skipped directory. Matches.
Cluster id disagrees with a formatted directory Invalid cluster.id in: <dir>/meta.properties. Expected <x>, but read <y>, raised as an uncaught exception, exit 1. The same message on stderr, exit 3. Matches the message. Kafka prints a stack trace because it does not catch the exception; krabka prints only the message.
Node id disagrees with a formatted directory Stored node id <stored> doesn't match previous node id <node.id> in <dir>/meta.properties. If you moved your data, make sure your configured node id matches. If you intend to create a new node, you should remove all data in your data directories., with or without --ignore-formatted, exit 1. The same message on stderr, exit 3. Nothing is written. Matches the message.
meta.properties file A Java properties file, written by Properties.store with an empty comment: a # line, a # line with java.util.Date.toString in the JVM’s time zone, then cluster.id, directory.id, node.id, and version=1, in key order on Java 18 and later. The reader is Properties.load. A version 1 file needs node.id and cluster.id, but not directory.id. The same bytes, with the date in UTC. The reader takes what Properties.load takes: comments, =, :, or white space as the separator, line continuations, and escapes. It refuses a version 0 file, and a cluster.id that is not a Kafka Uuid. Matches the bytes, except for the time zone of the date. UTC is the default zone of Kafka’s container images. A Kafka tool reads the file of a krabka directory, and the broker reads the file of a Kafka directory. Only a ZooKeeper broker writes version 0. The broker holds the cluster id as 16 bytes (see the cluster id form). The read errors are Kafka’s messages.
Bootstrap metadata Kafka 4.3.1 writes bootstrap.checkpoint, a KRaft snapshot of the bootstrap records, in every directory, and its broker reads the file only from metadata.log.dir. Kafka trunk writes the bootstrap records into the snapshot of each metadata directory, and a data directory gets only meta.properties (DirectoryType.isMetadataDirectory in Formatter.doFormat). bootstrap.json, a readable manifest, and bootstrap.records.bin, the same records as length-prefixed serde_wincode payloads, in the metadata log directory only. A data directory gets only meta.properties. Kafka has no equivalent of these two files. A controller loads the bootstrap records from bootstrap.records.bin in its metadata log directory, and the active controller writes them to the metadata log, as Kafka’s does. A broker never writes them. bootstrap.json lets an operator inspect them. The directories that get them match Kafka trunk. A copy in a data directory has no reader, in Kafka 4.3.1 or in krabka.
Dynamic quorum snapshot __cluster_metadata-0/00000000000000000000-0000000000.checkpoint in the metadata log directory, with the KRaftVersionRecord and the VotersRecord. The same file, with the same KRaftVersionRecord and VotersRecord, and the bootstrap records after them, as in Kafka trunk. A controller takes its bootstrap records from this file in place of bootstrap.records.bin, and does not apply them to its image. The active controller writes them to the metadata log, as Kafka’s QuorumController.handleLoadBootstrap keeps them for its activation. Matches the path, which is Kafka’s metadata partition directory layout, and the use of the records.
Future log directory krabka-format writes none. The Kafka broker names the directory of a KIP-113 replica move <topic>-<partition>.<32-hex-uuid>-future (LocalLog.logFutureDirName). The krabka broker names it <topic>-<partition>-future, with no unique id. A partition has at most one move, and only the krabka broker reads the name. A JVM broker cannot parse the name of an interrupted krabka move.
Clean-shutdown marker krabka-format writes none. The Kafka broker leaves .kafka_cleanshutdown in each log directory, a JSON object with the broker epoch (CleanShutdownFileHandler). The krabka broker leaves the same file, .kafka_cleanshutdown with Kafka’s JSON {"version":0,"brokerEpoch":N}, but only in its --log-dir. The name and bytes match Kafka’s. The rule is Kafka’s previousBrokerEpoch rule. A marker that is missing, not readable, or of a version other than 0 makes the restart unclean, which is the safe result and what Kafka’s CleanShutdownFileHandler.read does.
Write order Bootstrap files, then meta.properties, written to meta.properties.tmp, synced, and renamed, and then the directory is synced. Checkpoint, bootstrap files, then meta.properties, written the same way. A rerun removes what an interrupted run left. Matches the marker-last order and the file names. The cleanup is what lets a rerun succeed without an rm -rf.
Progress output Bootstrap metadata: ..., then Formatting <type> <dir> with metadata.version <v>. per directory, in the order of a HashMap. Formatting <type> <dir> with metadata.version <v>. per directory, the metadata log directory first and then the --log-dir entries in command-line order. Then a summary line with the cluster id. The directory lines match. Kafka’s Bootstrap metadata line prints Java toString output, which has no krabka equivalent. Kafka gives the directory lines no fixed order, so krabka uses one that a test can pin.
Exit codes 0 on success, 1 on every failure. 0 on success; 2 for an iteration count below 4096 or a command line that does not parse; 3 for a formatted, foreign, or inconsistent directory; 4 for a write failure or an invalid voter set; 5 for an invalid feature or quorum mode. An orchestrator acts on the cause. Exit 3 means an operator has to look at the disk, and exit 4 can be a fault that a retry fixes. The codes predate this document, and scripts depend on them.

The cluster id form

Kafka’s Uuid.toString is unpadded base64url of the 16 bytes. Uuid.fromString refuses a string longer than 24 characters and a string that does not decode to 16 bytes, but it accepts a trailing == and ignores the unused low bits of the last character. krabka-format accepts the same strings, so AQIDBAUGBwgJCgsMDQ4PEB and AQIDBAUGBwgJCgsMDQ4PEA== both name the cluster AQIDBAUGBwgJCgsMDQ4PEA. The hyphenated form is also accepted on the command line, because krabka’s earlier releases printed it, but it is never written. The broker reads cluster.id and directory.id in the base64 form only, as Kafka’s Uuid.fromString reads directory.id. A directory that a 0.x krabka formatted, with meta.properties.json and no meta.properties, is formatted again rather than read. Data written before 1.0.0 gets no compatibility promise. See Persisted formats.

Re-running after a failure

A run writes meta.properties last, and publishes it with a rename, so its presence means that a run finished in that directory. A directory with no meta.properties whose only contents are the files a run writes before it – bootstrap.json, bootstrap.records.bin, meta.properties.tmp, and the offset-zero checkpoint – is the result of an interrupted run. The next run removes those files and formats the directory. A directory that holds anything else is refused, so the cleanup cannot delete data that krabka-format did not write.

Integration

On every boot the broker reads the meta.properties of its metadata log directory, --metadata-log-dir or --log-dir without it, and of every data directory. It reads the files with krabka_format::MetaProperties, the reader and writer of krabka-format, and applies the checks of Kafka’s KafkaRaftServer.initializeLogDirs with Kafka’s messages. The files agree on the cluster id. Each file records the node id of --broker-id, and a directory of another node stops the broker with the Stored node id message above. The metadata log directory has to be formatted. A file without a directory.id gets one. Where the broker differs from Kafka, a data directory with no meta.properties is a disk that was added after the format: the broker writes Kafka’s file into it, with the node’s ids and a new directory id, where Kafka refuses to start. A controller reads the bootstrap records from the metadata log directory only. A broker-only node does not read them: it applies the records that the active controller writes to the metadata log. krabka-restore formats its target through krabka_format::run_from_args_with_records.

Kafka / KIP Compliance

  • KIP-853: --standalone, --initial-controllers, and --no-initial-controllers, and the directory id of the voter directory.
  • KIP-858: a directory id per log directory, never one of the reserved ids.
  • KIP-1022: --release-version and --feature.

Testing

crates/format/tests/kafka_storage_parity.rs runs the binary for each row that says Matches: the id round trip for each accepted and refused input form, a multi-directory format, a format with --metadata-log-dir, --ignore-formatted over a mixed set and its rerun, a run interrupted after each file it writes, and the bytes of each meta.properties against the bytes Kafka writes for the same ids. The interruption uses KRABKA_FORMAT_FAIL_AFTER, a test seam that makes a run fail after it writes the named file. The unit tests of crates/format/src/meta_properties.rs pin the reader against the files of each Java version and against Kafka’s message for each bad file, and the tests of crates/broker/src/bootstrap.rs pin the boot checks.