Crate krabka_remote_storage_topic 

Topic-backed RemoteLogMetadataManager for Krabka, part of the KIP-405 tiered-storage stack.

This crate ships TopicBasedRemoteLogMetadataManager, the production replacement for krabka_remote_storage::InmemoryRemoteLogMetadataManager. The manager appends remote-segment lifecycle events, which are add, update, and partition-delete, to an event log. In production that log is the __remote_log_metadata Kafka topic. Every broker rebuilds its local cache from the same log. After a restart, a broker re-reads the topic from offset 0 and re-applies the full history to recover its cache.

What this crate provides

• TopicBasedRemoteLogMetadataManager — the RemoteLogMetadataManager implementation.
• MetadataEventLog — the publish/subscribe seam between the manager and the underlying durable transport.
• InProcessMetadataEventLog — an in-memory fixture for unit tests and for modelling the multi-broker case without bringing up a real cluster (multiple managers cloned from the same Arc observe each other’s writes).
• MetadataEvent and the serde module — the on-wire binary codec for the three event variants.
• metadata_partition_for — the TopicIdPartition → metadata-topic-partition hash.
• KafkaMetadataEventLog — the production MetadataEventLog adapter that wires the trait to krabka_client_producer / [krabka_client_core] / [krabka_client_admin], persisting events in the __remote_log_metadata topic. Reads use manual per-partition Fetch loops over krabka_client_core, with no consumer group.
• MetadataPartitionReadiness — the per-metadata-partition readiness sample that TopicBasedRemoteLogMetadataManager::metadata_partition_readiness reports, which the broker exports as gauges.
• SwappableRlmm — the hot-swap facade the broker boots behind so it can start on the fail-closed NotReadyRlmm and upgrade to the topic-backed manager once its listener is serving. Broker::start selects the topic-backed manager when the [remote_storage.kafka_metadata] config section is present and in_memory is not set to true.

Operational boundaries

The metadata topic is an append-only event log created with delete cleanup and infinite retention. The manager maintains a local snapshot cache so restarts can resume from committed per-partition offsets instead of replaying the full topic every time. It does not use a Kafka consumer group or broker offset commits. The broker drives assignments explicitly with TopicBasedRemoteLogMetadataManager::reconcile_assignment. Internal Kafka clients use plaintext loopback by default, or the TLS/SASL settings supplied through KafkaMetadataLogConfig::security.

In-process manager for tests and local tools

use {
    krabka_remote_storage_topic::{
        InProcessMetadataEventLog, TopicBasedRemoteLogMetadataManager,
    },
    std::{path::PathBuf, time::Duration},
};

let event_log = InProcessMetadataEventLog::new(16);
let manager = TopicBasedRemoteLogMetadataManager::start(
    event_log,
    tokio::runtime::Handle::current(),
    PathBuf::from("/var/lib/krabka/rlmm-cache"),
    Duration::from_secs(30),
)?;

manager.reconcile_assignment(&[0, 1]).await;

Re-exports

pub use error::CodecError;

pub use error::MetadataLogError;

pub use error::SnapshotError;

pub use kafka_log::DEFAULT_METADATA_EVENT_QUEUE_CAPACITY;

pub use kafka_log::DEFAULT_METADATA_FETCH_MAX_BYTES;

pub use kafka_log::DEFAULT_METADATA_FETCH_MAX_WAIT;

pub use kafka_log::DEFAULT_METADATA_FETCH_RETRY_BACKOFF;

pub use kafka_log::DEFAULT_METADATA_TOPIC_CREATE_TIMEOUT;

pub use kafka_log::DEFAULT_MIN_ISR;

pub use kafka_log::DEFAULT_NUM_PARTITIONS;

pub use kafka_log::DEFAULT_REPLICATION;

pub use kafka_log::KafkaMetadataEventLog;

pub use kafka_log::KafkaMetadataLogConfig;

pub use kafka_log::METADATA_TOPIC;

pub use kafka_log::MetadataEventQueueCapacity;

pub use log::AssignmentHandle;

pub use log::InProcessMetadataEventLog;

pub use log::KeyedEvent;

pub use log::MetadataEventLog;

pub use log::MetadataEventRecord;

pub use log::MetadataEventStream;

pub use log::PartitionStart;

pub use log::RangeVisitor;

pub use manager::MetadataPartitionReadiness;

pub use manager::TopicBasedRemoteLogMetadataManager;

pub use not_ready::NotReadyRlmm;

pub use partitioning::metadata_partition_for;

pub use partitioning::metadata_partitions_for;

pub use serde::MetadataEvent;

pub use snapshot::SNAPSHOT_FILE_NAME;

pub use snapshot::SNAPSHOT_FORMAT_VERSION;

pub use snapshot::Snapshot;

pub use swappable::SwappableRlmm;

Modules

error

Errors surfaced by the topic-backed RemoteLogMetadataManager and the underlying MetadataEventLog.

kafka_log

KafkaMetadataEventLog: the production MetadataEventLog adapter that persists events in the internal __remote_log_metadata Kafka topic.

log

MetadataEventLog: the publish/subscribe seam between the TopicBasedRemoteLogMetadataManager and the underlying durable event store.

manager

TopicBasedRemoteLogMetadataManager: the production RemoteLogMetadataManager implementation backed by a publish and subscribe MetadataEventLog.

not_ready

NotReadyRlmm: a fail-closed RemoteLogMetadataManager placeholder.

partitioning

Deterministic TopicIdPartition → metadata-topic-partition mapping.

serde

Binary wire format for __remote_log_metadata events.

snapshot

On-disk RLMM snapshot: a versioned envelope wrapping a RlmmCacheDump plus the per-metadata-partition committed offsets.

swappable

SwappableRlmm: a RemoteLogMetadataManager facade whose backing implementation can be replaced atomically after construction.