Module kafka_log 

KafkaMetadataEventLog: the production MetadataEventLog adapter that persists events in the internal __remote_log_metadata Kafka topic.

Writes flow through a krabka_client_producer::Producer with explicit per-record partition pinning. Reads come back through one cancellable manual-Fetch task per assigned partition. Each task drives its own dedicated [krabka_client_core::Connection] and emits MetadataEventRecords into a shared mpsc. There is no consumer group and no broker-side offset commit. The RLMM owns the read position. The manager assigns all partitions from offset 0 today, then resumes from snapshot offsets and restricts the consumed set.

A dedicated connection per partition is necessary because the broker is serial per-connection. A long-max_wait_ms fetch would head-of-line-block any other RPC that shares the socket.

Topic provisioning runs once at KafkaMetadataEventLog::start through the [krabka_client_admin::AdminClient]. It reuses an existing topic, and the topic’s actual partition count then overrides the configured num_partitions. It creates an absent topic with the configured cleanup policy and retention.ms=-1. The same admin round-trip surfaces the topic’s Uuid, which the manual Fetch path needs, because Fetch v≥13 carries topic_id and not the name.

One ListOffsets(timestamp=-1) over the raw [krabka_client_core::Client] pulls the high-water marks, rather than a consumer. MetadataEventLog::high_water_marks therefore does not need any fetch task to have made progress.

KafkaMetadataEventLog::open_read_only opens the same log without a producer and without provisioning. It sends no InitProducerId and no Produce, so a principal that holds only READ and DESCRIBE on the topic can use it. MetadataEventLog::visit_range then reads a bounded offset range from the partition leader one page at a time, and follows the leader when it moves. The range module holds that loop.

Structs

KafkaMetadataEventLog

Production MetadataEventLog backed by an internal Kafka topic.

KafkaMetadataLogConfig

Construction-time configuration for KafkaMetadataEventLog.

MetadataEventQueueCapacity

Positive capacity of the shared metadata-event delivery queue.

Constants

DEFAULT_METADATA_EVENT_QUEUE_CAPACITY

Default capacity of the shared metadata-event delivery queue.

DEFAULT_METADATA_FETCH_MAX_BYTES

Per-partition budget for the metadata Fetch. Metadata events are small, so one mebibyte is many thousands of them per round-trip.

DEFAULT_METADATA_FETCH_MAX_WAIT

max_wait_ms for the per-partition metadata Fetch. It is long enough that an idle partition costs one RPC per interval rather than a spin, and short enough that cancellation on reassignment is prompt.

DEFAULT_METADATA_FETCH_RETRY_BACKOFF

Pause before a retry of a failed metadata Fetch, so a broker that is down does not turn the fetch loop into a busy spin.

DEFAULT_METADATA_TOPIC_CREATE_TIMEOUT

How long CreateTopics may take to provision __remote_log_metadata before the broker abandons the round-trip.

DEFAULT_MIN_ISR

Default min.insync.replicas for __remote_log_metadata, matching Apache Kafka’s remote.log.metadata.topic.min.isr (TopicBasedRemoteLogMetadataManagerConfig.DEFAULT_REMOTE_LOG_METADATA_TOPIC_MIN_ISR).

DEFAULT_NUM_PARTITIONS

Default partition count for __remote_log_metadata, matching Apache Kafka’s remote.log.metadata.topic.num.partitions.

DEFAULT_REPLICATION

Default replication factor for __remote_log_metadata, matching Apache Kafka’s remote.log.metadata.topic.replication.factor.

METADATA_TOPIC

Default name of the internal metadata topic.