Crate krabka_log 

Byte-compatible reader/writer for Apache Kafka’s on-disk log format.

This crate is the append-only storage layer that the Krabka broker uses. It reads and writes Kafka 4.x’s on-disk log format byte-for-byte: 20-digit zero-padded segment filenames, sparse .index and .timeindex files, and append-only .log files that hold [krabka_protocol::records::RecordBatch] v2 streams.

What this crate does

• Open and recover existing log directories.
• Append RecordBatches to the active segment.
• Read sequentially from an absolute offset.
• Truncate the log to an offset, for replication and leader election.
• Time-based and size-based retention.
• Deliver-at-time visibility, where a batch becomes readable only once its activation time has passed.

Scope and boundaries

This crate is the byte-compatible segment and index layer. It exposes append, read, truncation, leader-epoch checkpoint, transaction-index, retention, and compaction primitives over a single log directory. krabka-broker applies broker-level policy above this storage layer. That policy covers topic configuration, leader and follower ownership, tiered-storage scheduling, transaction visibility rules, and write serialization.

Quick start

use {
    krabka_ids::Offset,
    krabka_log::{Log, LogConfig},
    krabka_protocol::records::RecordBatch,
    krabka_units::prelude::mebibytes,
};

let mut log = Log::open("/var/kafka/my-topic-0", LogConfig::default()).unwrap();
let mut batch = RecordBatch::default();
// ... fill the batch ...
let assigned_offset = log.append(&mut batch).unwrap();

let out = log.read(Offset(0), mebibytes(1)).unwrap();

Exporting a segment

use krabka_log::{Log, LogConfig};

let mut log = Log::open("/var/kafka/my-topic-0", LogConfig::default())?;
for segment in log.tierable_segments() {
    println!(
        "segment {} starts at offset {}",
        segment.log_path.display(),
        segment.base_offset
    );
}

Modules

name

Names inside a Kafka log directory: the per-partition directory, and the segment files in it.

topic_name

The topic-name rules of Apache Kafka’s org.apache.kafka.common.internals.Topic.

Structs

AbortedTxn

ActiveProducer

One producer as DescribeProducers reports it.

ActiveSegmentExport

The active segment as tiered local retention measures it (KIP-405).

BarrierMarker

The contents of one barrier marker.

BuiltCompaction

A built CompactionJob, for Log::finish_compaction to install. Dropping it instead deletes the files the build wrote.

CompactionContext

Inputs to one Log::compact pass that depend on broker-side state.

CompactionJob

A compaction pass taken from a log by Log::prepare_compaction, ready to build without the log.

DeliveryAdvance

Result of Log::advance_delivery_watermark.

DueDeletions

Retired segment files whose deletion delay has passed, taken out of the log by Log::maintain_deferring_deletions.

EpochEntry

One leader epoch and the offset where it begins.

FailedDeletion

A DueDeletions::delete that stopped at a failure.

FileRegion

A [offset, offset+len) byte range within a segment .log file.

LeaderEpoch

A partition leader epoch. KIP-320 wire type: int32.

LeaderEpochCheckpoint

Log

A Kafka-format log: a sorted collection of Segments plus a single active segment that accepts appends.

LogConfig

Tunables for Log behavior.

Offset

A record offset within a topic partition’s log. KIP wire type: int64.

ProducerBatchMetadata

One data batch a producer appended: Kafka’s BatchMetadata.

ProducerId

A Kafka idempotent or transactional producer id. KIP-98 wire type: int64.

ProducerSnapshotEntry

Durable producer metadata stored in Kafka’s .snapshot sidecar format.

RawRead

Verbatim, decode-free output of Log::read_raw.

RawReadDesc

Descriptor form of Log::read_raw for the zero-copy sendfile fetch path, Increments D + E.

RawReadPlan

A raw read planned under the log lock, to run without it.

RawSegmentDesc

Descriptor form of Segment::read_raw for the zero-copy fetch path, Increments D + E.

RawSegmentRead

Verbatim, decode-free output of Segment::read_raw.

ReadOutput

Result of Log::read: the absolute offset of the first batch returned and the batches themselves.

RecoveredProducer

One producer’s state as the log holds it: its snapshot entry, whose last batch is the producer’s last one, and the batches the producer appended before that one at the same epoch, oldest first.

RemoteTierFlags

KIP-950’s two tiered-storage disablement controls.

Segment

SegmentExport

A sealed segment described for tiered-storage offload (KIP-405).

StampEntry

One stamped offset range. The inclusive offsets [base_offset, last_offset] all carry stamp.

StampIndex

TransactionalBatch

One batch as the transaction checks see it.

TxnIndex

VerbatimBatch

A batch that the log appends verbatim, with no re-encode.

VerificationGuard

A token for one verification of one producer on one partition.

Enums

CleanupPolicy

Per-topic policy for what to do with old log segments.

CompactionOutcome

What Log::finish_compaction did with a built pass.

DeliveryPolicy

Per-topic policy for when a durable record becomes visible to consumers.

FilteredBatch

What filter_batch made of a batch.

LogError

Errors returned by Log and Segment.

ScheduleOrder

Whether a scheduled partition’s delivery times must not run backwards.

SegmentAllocation

How a new segment’s .log file gets its disk blocks: Kafka’s preallocate.

SnapshotDecodeError

Why decode rejected a producer-state .snapshot. Each variant names the check that failed and carries what that check saw; its Display is the reason the log reports in LogError::Corrupt.

TransactionAppendRefusal

Why a transactional produce may not start or continue its append.

Constants

BARRIER_CONTROL_TYPE

Control-record type of a krabka barrier marker.

DEFAULT_MAX_MESSAGE_SIZE

Kafka’s max.message.bytes default: 1 MiB of records plus the 12-byte Records.LOG_OVERHEAD that prefixes every batch on the wire.

DEFAULT_READ_AHEAD_MAX

Default cap on how far past its own window a verbatim read asks the kernel to read ahead.

DEFAULT_TAIL_CACHE_SIZE

Default for LogConfig::tail_cache_size.

DEFAULT_WRITEBACK_WINDOW

Default for LogConfig::writeback_window.

NUM_BATCHES_TO_RETAIN

Kafka’s ProducerStateEntry.NUM_BATCHES_TO_RETAIN: the number of a producer’s most recent batches whose retry answers as a duplicate.

Traits

StampSource

A source of monotonic internal stamps folded into a partition’s .stampindex.

Functions

batch_is_deliverable

Apply the shared scheduled-delivery visibility rule to one batch.

decode_producer_snapshot

Decode a Kafka v1 producer-state .snapshot taken at snapshot_offset, the exclusive log frontier its entries must lie strictly before.

epoch_and_offset_for_entries

Resolve a requested leader epoch to (found_epoch, end_offset), exactly as Kafka’s LeaderEpochFileCache.endOffsetFor does (see kafka_end_offset_for for the case table).

epoch_ms

now in epoch milliseconds. An instant before the epoch reads as 0, and one past i64::MAX milliseconds saturates there.

filter_batch

Drop the records of batch for which keep returns false.

parse_barrier_marker

Read the contents of a barrier marker back out of its record.

write_file_atomic

Replace path with bytes atomically and durably through real file I/O: write tmp, sync it, rename it over path, then sync the parent directory.