Struct Segment 

pub struct Segment { /* private fields */ }

Implementations

impl Segment

pub fn append( &mut self, batch: &RecordBatch, index_interval: ByteSize, ) -> Result<u64, LogError>

Append a record batch and return the byte position where the batch starts.

Side effects:

• Updates log_size, max_timestamp, and last_offset.
• Adds sparse index entries when the byte count since the last entry exceeds index_interval, which Kafka’s LogSegment.append does not do for the first batch either.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

pub fn append_verbatim( &mut self, bytes: &[u8], base_offset: Offset, last_offset_delta: i32, max_timestamp: i64, leader_epoch: LeaderEpoch, index_interval: ByteSize, ) -> Result<u64, LogError>

Append a batch verbatim and write the producer’s exact wire bytes.

This method does not decode, re-encode, recompress, or recompute the CRC. bytes is the producer’s verbatim v2 batch, which the caller has already CRC-validated through the borrowed header-only path. This method patches only base_offset, bytes 0..8, and partition_leader_epoch, bytes 12..16, into a writable copy, then writes those bytes. Both fields sit outside the CRC-covered region. The stored CRC stays byte-identical to the producer’s CRC, because no CRC-covered byte changes.

base_offset, last_offset_delta, max_timestamp, and leader_epoch come from the caller’s borrowed header read. This method updates the segment side effects, log_size, last_offset, max_timestamp, and the sparse index, exactly as Segment::append does.

It returns the byte position where the batch starts.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

impl Segment

pub fn seal(&mut self) -> Result<(), LogError>

Mark this segment as sealed. No more appends.

Sealing first writes the segment’s final time-index entry, Kafka’s LogSegment.onBecomeInactiveSegment: the sparse index lags the writes, so without it a reopened segment would not learn the timestamp of the batches after its last index point from the index. A segment that wrote through O_DIRECT then goes back to the page cache, which cuts the padding past its last batch, and what is left of its block reservation is given back.

Errors

Returns an error when the time-index entry cannot be written or its offset overflows the index range, or when the padding cannot be cut. The segment stays open then.

pub fn seal_at(&mut self, last: Offset)

Seal a segment loaded without a scan and set its last_offset to last.

Callers pass next_segment.base_offset - 1, the highest offset this sealed segment can hold. The no-scan loader leaves last_offset = base_offset - 1 because it does not scan the .log. Without this fix, a sealed segment recovered on Log::open reports that stale last_offset. Log::read_raw skips any segment whose last_offset() < fetch_offset, so it would skip the first sealed segment after a restart and serve a later segment’s base offset. That creates an offset gap, and a follower that fetches at 0 then loops on the resulting append mismatch.

pub fn flush(&mut self) -> Result<(), LogError>

Force-sync everything to disk.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

pub fn truncate_to_relative(&mut self, rel: u32) -> Result<(), LogError>

Truncate the .log file and the indexes so that no batch at relative_offset >= rel remains. Log::truncate_to uses this method. The segment stays unsealed.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

impl Segment

pub fn create(dir: &Path, base_offset: Offset) -> Result<Self, LogError>

Create a fresh active segment at the given base offset. This fails if the .log file already exists.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

pub fn open_active( dir: &Path, base_offset: Offset, validate: bool, ) -> Result<Self, LogError>

Open as the active segment.

When validate is true, this method scans from byte zero to EOF. It truncates a partial or invalid trailing batch, rebuilds both sparse indexes, and updates last_offset and max_timestamp from the same maximal valid prefix.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

pub fn open(dir: &Path, base_offset: Offset) -> Result<Self, LogError>

Open an existing segment for reading and refuse further appends.

Scans the batches to restore the last offset and maximum timestamp. A corrupt or incomplete batch returns an error without truncating the file. Use Segment::open_active to recover a writable segment.

Errors

Returns an error when the segment or its indexes cannot be opened, its batches cannot be read, or its offset chain or batch bytes are corrupt.

impl Segment

pub fn read( &self, offset: Offset, max_size: ByteSize, ) -> Result<Vec<RecordBatch>, LogError>

Read batches from offset or just before it, up to about max_bytes of .log data. The result is an empty Vec when offset is past last_offset.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

impl Segment

pub fn read_raw( &self, fetch_offset: Offset, limit_offset: Offset, max_size: ByteSize, ) -> Result<RawSegmentRead, LogError>

Read a contiguous run of complete, verbatim record-batch bytes.

The run starts at the batch that contains fetch_offset. It includes only batches whose base_offset < limit_offset, up to about max_bytes, and always at least one batch. That last rule is Kafka’s anti-stall rule. This method decodes no records. It reads only the fixed batch headers to find the boundaries.

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

impl Segment

pub fn read_raw_desc( &self, fetch_offset: Offset, limit_offset: Offset, max_size: ByteSize, ) -> Result<RawSegmentDesc, LogError>

Descriptor variant of Segment::read_raw for the zero-copy sendfile fetch path.

This method runs the same boundary walk and selects the identical [start_pos+range_start, start_pos+range_end) byte range that read_raw would have sliced. It returns a krabka_protocol::records::FileRegion descriptor instead of a pread of the payload into an owned Bytes.

The walk reads bounded windows to amortize header lookups and uses the header’s batch_length to find boundaries. It returns a file region instead of building an owned copy of the selected payload. The region is byte-identical to the bytes of read_raw for the same (fetch_offset, limit_offset, max_bytes).

Errors

Returns an error when log I/O fails, a record or index is corrupt, or the requested offset violates the segment state.

impl Segment

pub fn read_view( &self, fetch_offset: Offset, cold_before: Option<u64>, ) -> Result<SegmentReadView, LogError>

A view of this segment for a read from fetch_offset, to run after the log lock is released. cold_before is the log’s verdict on where a fetch of this segment turns cold, taken now because it needs the log.

Errors

Returns an error when fetch_offset is too far past the segment for its index, which no read of the segment could use either.

impl Segment

pub fn offset_for_timestamp(&self, target_ts: i64) -> Option<(Offset, i64)>

Absolute offset and record timestamp of the first record in this segment whose timestamp is >= target_ts.

This method takes a floor position from the sparse time index, then scans .log batches forward. The index is sparse, so an exact answer needs that scan after the index lookup. This matches Kafka’s LogSegment.findOffsetByTimestamp. The result is None when no record in this segment qualifies.

The scan applies no max.decompressed.message.bytes: this is the lookup the log’s own bookkeeping uses, to read a segment’s first timestamp. A ListOffsets answer goes through Segment::offset_for_timestamp_with_window, which takes the limit.

pub fn restore_max_timestamp(&mut self) -> Result<(), LogError>

Restore max_timestamp for a segment loaded without a scan.

The no-scan loader leaves the field at its unknown sentinel. Retention compares that sentinel against the age cutoff, so without this call every reopened segment looks older than any window and the first Log::tick after a restart deletes all of them.

Kafka’s LogSegment reads maxTimestampSoFar back from the time index, and this method starts there. The sparse index lags the writes, though: its newest entry holds the running maximum as of the last indexed batch, and every batch appended after it is unaccounted for. The method therefore also walks the batch headers from the newest offset-index entry to the end of the file. That walk is bounded by index_interval plus one batch, and it reads no record body.

Errors

Returns LogError::Io when the .log file cannot be read.

impl Segment

pub fn base_offset(&self) -> Offset

Absolute offset of the first record this segment can hold.

pub fn txn_index_path(&self) -> PathBuf

Path to this segment’s .txnindex file (may not exist yet).

pub fn stamp_index_path(&self) -> PathBuf

Path to this segment’s .stampindex sidecar (may not exist yet).

pub fn leader_epoch_checkpoint_path(&self) -> PathBuf

Path to the per-partition .leader-epoch-checkpoint file in this segment’s directory. All segments in a partition share the checkpoint, and epoch history accumulates over the log’s lifetime.

pub fn last_offset(&self) -> Offset

Highest absolute offset (inclusive) of any batch appended to this segment. Returns base_offset - 1 for an empty segment.

pub fn size(&self) -> ByteSize

Current .log file size.

The field itself stays a raw u64, because it is a file position and goes directly into a pread or seek argument. It converts here, where the value becomes a magnitude that the roll and retention policies compare.

pub fn max_timestamp(&self) -> i64

Highest timestamp observed across all batches in this segment. Returns i64::MIN for an empty segment, and can also read -1 (NO_TIMESTAMP) when every batch in the segment carries no timestamp. See Segment::last_modified for Kafka’s fallback for both cases.

pub fn last_modified(&self) -> Option<SystemTime>

The .log file’s last-modified time, as Kafka’s LogSegment. lastModified reads it. Time retention falls back to this when the segment carries no record timestamp of its own (max_timestamp() is i64::MIN or -1), the same as UnifiedLog’s largestRecordTimestamp().orElse(lastModified()).

None when the filesystem cannot report it, which retention treats as “unknown, not overdue” rather than as license to guess.

pub fn is_sealed(&self) -> bool

true once Segment::seal has sealed the segment. Sealed segments reject appends.

pub fn dir(&self) -> &Path

Directory that holds this segment’s .log, .index, and .timeindex files. The compactor uses it to read the .log file directly and to avoid the Segment::read path. That path depends on the in-memory last_offset, which is stale for sealed segments loaded through the log’s no-scan loader.

Trait Implementations

impl Debug for Segment

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.