Release performance against Kafka 4.3.1 — 2026-09-30

This is a local throughput, CPU, and memory experiment on one shared Linux host. All reported comparisons use the same Kafka 4.3.1 client, broker resource limits, payload, acknowledgments, replication contract, and number of repetitions. Medians summarize three independent topics; the accompanying CSV retains each run.

The tables are the archived benchmark results for the exact sources and binary hashes below. Publication rebased the changes onto broker a5142413 and protocol a6684fca, which had advanced while the benchmark was running. The broker pins protocol commit 8e385ae182b89dca6c6b119ab865142a6a57d0ea from krabka-protocol CRC follow-up. These archived numbers do not claim a rerun of the rebased PR revision.

Changes and evidence

The first CPU profile showed repeated first-batch CRC verification, decompression, and record allocation through should_roll_for_incoming. Cache the first record’s timestamp per segment, invalidate it on rollback/truncation, and recover it lazily on reopen. This preserves CreateTime/LogAppendTime behavior and negative timestamp deltas. The next profile confirmed that repeated owned batch decoding disappeared. It also identified request guards cloning/dropping every metrics handle; those guards now borrow the metrics for their existing lifetime.

The final random-payload/LZ4 CPU profile confirms the repeated roll-check decoding is gone. It attributes about 28% of sampled time to the sendfile path, 5% to the new CRC backend, and 5% cumulatively to record parsing. For zero-filled compressed traffic, decompression and record validation remain much larger costs. CPU sample shares are qualitative attribution, not instruction counts or a demonstrated universal throughput ceiling.

--config=release selects -c opt, thin LTO, and one codegen unit. Pilot runs of both source fixes with ordinary opt versus thin LTO had medians of 569,003 versus 643,790 records/s, and 17.894 versus 17.495 broker CPU seconds for 10 million 1 KiB random records. The small CPU difference is inconclusive; binary size fell from 80,446,224 to 57548696 bytes. These pilots are separate from the matched matrix below.

The workload accepts optional compression and zeros|random arguments. A seeded 4 MiB random pool prevents mostly zero payloads from masquerading as incompressible traffic. Every record retains its timestamp and unique sequence number. Existing seven-argument invocations retain their previous LZ4/zeros behavior.

Contiguous CRC follow-up

After the archived broker comparison, the borrowed batch validator and decoder were changed to hash the contiguous covered header and raw body in one call. They avoid a separate 40-byte header CRC and seeded digest setup, without copying or allocating a combined buffer. CRC coverage remains bounded by batch_length.

A release microbenchmark compared protocol 0536491e with 8e385ae1 on this host, using thin LTO, one codegen unit, CPU affinity to core 2, and five alternating before/after trials. Each timed public API call checks one uncompressed batch containing one record. Medians in nanoseconds per batch:

These are isolated API wall times, not whole-broker CPU or memory measurements. A direct CRC probe found negligible gains at 64 KiB and 1 MiB, where hashing dominates setup. The archived Kafka tables do not include this follow-up. Raw trials and probe sources are under crc-probe/ and record-probe/ in the local artifact directory below. The checked-in records benchmark now includes small validation cases, and a regression test verifies that a corrupt adjacent batch cannot affect the first batch’s CRC.

Contract and provenance

• Baseline broker commit: 401945732d9ac37b93de5c89d5e649ece18049b0.
• Kafka: apache/kafka:4.3.1, pinned as apache/kafka@sha256:77e3df9054047a88b520d0cc46e16696d3b22022e1d580aeccd2632df6532837.
• Native runtime: the repository’s Wolfi production image, immutable local image sha256:66999791b48d10e3ef53a5c3ccaf0b7d30389040138e5476392540ad43b7662e, with the measured binary mounted read-only. Initial Alpine/glibc launch failures were archived and contributed no measurements.
• Host: AMD EPYC 4344P, 8 physical/16 logical CPUs, about 61 GiB RAM, RAID-backed disk, loopback networking. No TLS, authentication, tiered storage, compaction, forced per-append fsync, restart, or failure injection in these runs.
• Each broker/controller: two physical cores, 2 CPU quota, 2 GiB memory, no swap. Kafka heap: -Xms1g -Xmx1g, except the explicitly labeled smaller-heap check.
• RF1: one combined broker/controller on cores 2–3; client on cores 0–1 and 4–7. RF3: three combined broker/controllers on cores 2–3, 4–5, and 6–7; client on cores 0–1 plus their SMT siblings 8–9.
• Each topic: 12 partitions, RF1/minISR1 or RF3/minISR2, acks=all, idempotence, 65,536-byte producer batches, 5 ms linger. Clients use OpenJDK 25.0.4.1 and the exact Kafka 4.3.1 jars for both brokers. Admin JVMs run outside broker cgroups.
• RF1 warm-up: 3 million records; RF3: 1 million records. Each measured topic is fresh, while the broker remains alive across the same ordered case sequence. RF1 samples: 10 million records for 1 KiB/100 B LZ4, 5 million uncompressed, 30,000 for 100 KiB, or 600,000 at 20k/s for 30 seconds.
• CPU is the broker cgroup’s cpu.stat usage delta, including system CPU; RF3 adds all three brokers. CPU per record uses acknowledged records, not replicas. Monitoring includes client initialization/closing, whereas workload throughput and latency exclude that initialization. Short unlimited cases consequently have a larger fixed overhead and more variance.
• Peak process RSS, anonymous cgroup memory, and working set are sampled every 250 ms. Working set is memory.current - inactive_file; RSS excludes most disk page cache. These are observed peaks, not hard bounds. Profiling runs are excluded from memory comparisons because pprof allocates additional buffers.
• Every completed workload consumed exactly its acknowledged count with zero producer errors and zero duplicate sequences. This validates delivery in the experiment; it is not a crash-durability qualification.

Single broker, RF1

Local protocol optimization

The table above uses the benchmark baseline’s original protocol dependency. The following variant also changes the sibling krabka-protocol checkout: use the runtime-dispatched crc-fast CRC-32C backend, preserve finalized seeded CRC semantics, and inline four existing varint readers. CRC equality was checked against the previous independent implementation for arbitrary seeds, unaligned buffers, and chained chunks. A standalone 64 KiB CRC microbenchmark improved from 7.38 to 63.4 GiB/s on this AVX-512 host; whole-broker gains are much smaller. The compressed-record profile’s sampled varint share fell from about 29% to 13%; inlining changes attribution, so that percentage is not a whole-broker CPU gain.

The measured binary used a local two-repository variant. The protocol edits were in /home/matt/.codex/worktrees/782f/krabka-protocol, branch codex/crc32c-performance, based on 753a8e2a6795dad438b587910afa05b43bf63e95. The implementation is now published in protocol PR #48 and pinned by this broker PR. An ordinary release build includes both repositories’ optimizations. Replaying the exact historical measured binary uses the original source patches and both repository overrides listed in builds.json and below.

This variant improves original Krabka’s 1 KiB random throughput by about 35% and lowers its CPU per record by about 34%. Against Kafka, the 1 KiB random result is effectively tied within host variance. The other unlimited RF1 rows show about 19–33% greater throughput, but CPU is still about 33% higher for compressible records, 8% higher without compression, and 14% higher for large records. At 20k records/s it uses about 31% more CPU than Kafka. These are separate throughput and efficiency outcomes.

The random 100 B unlimited case is short; its headline rate is not evidence of a sustained broker saturation ceiling. See the longer concurrent-client check.

Interleaved large-record recheck

The CRC-only sequential matrix had a slow 100 KiB row. Three interleaved rounds with a fresh broker per variant, 5,000 warm-up records, and 60,000 measured 100 KiB random LZ4 records did not reproduce that regression:

CPU µs/record is lower-is-better; throughput is higher-is-better. RSS depends on heap policy and previous workloads: Kafka’s committed heap grows during the case sequence. The first 1 KiB row has substantially lower Kafka RSS than later rows. Highly compressible MiB/s measures logical payload bytes, not wire/disk bandwidth.

Three brokers, RF3/minISR2

CPU and memory below are aggregate cluster values. Throughput and latency are from the same client workload, with all replicas in the ISR before the run.

The local protocol variant, with the same RF3 contract:

The broker-only release is about 19% faster than Kafka in this RF3 run, while using about 11% more aggregate CPU per record. The local protocol variant is about 10% faster, with similar CPU to the broker-only variant. This difference does not establish a protocol throughput improvement for replication.

Repeated measurements and follow-up checks

Linux TCP packet batching experiment

Three interleaved rounds compared the local protocol variant with and without TCP_CORK around file-backed fetch responses, using a fresh broker each time. The socket is uncorked on success and I/O errors; cancellation already requires the caller to close a partially written response. TLS fallback and other OS paths are unchanged. Results below determine whether this additional code stays.

The TCP_CORK experiment was removed: median CPU gains were only 1.4% unlimited and 1.8% at 20k/s, below the materiality threshold and host variance. Its 9% median throughput difference came with overlapping 406k–686k versus 606k–692k ranges and an isolated p99 spike. The simpler previously validated release-final binary is used for follow-up comparisons.

A final isolated record-validation check compared the existing parser’s inline hint with forced inlining. Three interleaved samples of 64 KiB batches, with 100 B/1 KiB zero values and no compression/LZ4, showed no material improvement (median changes within about 1%). The last round had substantial host variance. No parser logic or additional inline attribute was retained. The corrected experiment and compiler output are retained in record-probe/forced-results.txt; the earlier duplicate-attribute trial is not evidence of an optimization.

Four concurrent producer/consumer pairs

Four independent topics each have 12 partitions (48 total). Each client writes and reads 5 million 1 KiB random, 40 million 100 B random, or 10 million 1 KiB zero records. Rate uses the total count divided by full wall time including client startup and shutdown, unlike the single-client matrix. Latency values are the worst client quantile, not a merged distribution; each client JSON is retained. All four JVMs share the same six client cores.

Fresh broker with Kafka 256 MiB heap

Both brokers start fresh, warm up with 3 million 1 KiB random records, then run three unlimited and three 20k/s topics. Kafka uses -Xms256m -Xmx256m; native broker settings remain the same. This check measures the effect of a smaller Kafka heap instead of attributing the entire memory gap to implementation.

Interpretation and limits

This host runs other workloads, and the measurements have visible variance. The small RF1 1 KiB throughput lead over Kafka should not be treated as a decisive win. Some single-client tests leave broker CPU below its limit; their throughput is an end-to-end workload rate, not a demonstrated broker saturation ceiling. CPU efficiency, throughput, and latency can move in different directions.

These results establish local behavior for these binaries and contracts. They are separate from the existing Kind performance qualification, partition-envelope results, remote CI, and release readiness.

The original native container inherited a 1,024-descriptor soft limit. Completed matrix runs stayed within it, but a later profiled random-record run after a 60-million-record profile exhausted descriptors and triggered the broker’s shutdown path. That incomplete run is excluded. All subsequent fresh-broker, packet-batching, concurrent, and small-heap runs explicitly use --ulimit nofile=131072:131072 for both brokers; saved /proc limits establish the setting. There was no OOM. This experiment does not qualify default-container behavior at an accumulated multi-topic file-descriptor limit.

Reproduction and retained artifacts

Build the current broker, including the published protocol pin, with:

bazel build --config=release --lockfile_mode=off --bes_backend= \
  //crates/broker:krabka-broker //crates/format:krabka-format
bazel run --config=release --lockfile_mode=off --bes_backend= //packaging:image_load

To replay the historical local variant on the archived broker source, add these to the build (the first directory preserves Bazel-generated BUILD files, and the second supplies its dependency metadata; a raw source checkout alone cannot replace those generated files):

--override_repository=rules_rs++crate+crates__github.com_krabka-io_krabka-protocol_753a8e2a=/home/matt/.cache/krabka-perf/782f-20260930/protocol-source-override
--override_repository=rules_rs++crate+crates__krabka-protocol-0.4.0=/home/matt/.cache/krabka-perf/782f-20260930/protocol-crate-override

Compile packaging/performance/BrokerPerformanceWorkload.java against Kafka 4.3.1 jars. Its complete argument contract is:

bootstrap topic group records bytes records_per_second timeout_seconds [compression] [zeros|random]

The exact local runner, container commands, configs, topic descriptions, resource time series, per-client JSON, profiles, build hashes, and validation logs are in /home/matt/.cache/krabka-perf/782f-20260930. run.py, matrix.py, rf3.py, and concurrent.py record the launch and workload commands. Paths in this experimental runner refer to this checkout; adapt ROOT/ART and the native runtime image ID for another host. builds.json records the source patch and binary SHA-256 hashes. Disposable broker record logs were removed after count/sequence validation to avoid disk exhaustion; configs, measurements, profiles, and binaries remain.

performance-kafka-4.3.1.csv contains each matched unprofiled run, including throughput, CPU seconds, peak RSS/anonymous/working-set memory, and p50/p95/p99. Profiles are retained in krabka-1k-profile, cache-1k-profile, and release-rf3-optimized-profile.

The following hashes identify the main native binaries independently of mutable Bazel outputs:

Validation

474 log unit tests, 4 restart tests, and all 4,620 broker unit tests passed with -c opt and the local protocol overrides after removing the packet-batching experiment. The experiment separately passed 4,621 broker unit tests. Broker and log Bazel Clippy targets passed. All 1,137 protocol unit tests and protocol Cargo Clippy (--all-targets --locked -D warnings) passed in the sibling checkout. Thin-LTO release builds completed the measured workloads. git diff --check and formatting passed. No remote CI conclusion is claimed.

Publication revalidation after the rebase and published protocol pin passed 4,634 broker unit tests, 474 log unit tests, and four restart tests through normal Bazel resolution, without repository overrides. Both repositories passed cargo clippy --workspace --all-targets --locked -- -D warnings; the rebased protocol companion passed 1,140 unit tests. Formatting and diff checks passed. The archived benchmark tables remain tied to their original binaries above.

The contiguous-CRC follow-up passed all 1,141 protocol unit tests and full workspace Clippy. The CI timeout repair leaves the Raft model and all state bounds intact: it gives only kraft_model_test a 900-second Bazel timeout, because its five cases exceeded the default 300-second limit on CI. All five cases passed locally with their pinned unique-state counts unchanged.

The same follow-up pin passed 4,634 broker unit tests, 474 log unit tests, four restart tests, full broker workspace Clippy, and formatting through normal dependency resolution. Updating newly yanked yoke-derive 0.8.3 to 0.8.4 passed all four cargo-deny checks.