Crate krabka_raft

Expand description

Metadata Raft quorum for Krabka.

krabka-raft runs a hand-rolled KIP-595 KRaft consensus engine, the kraft::KraftController, over Krabka’s storage (krabka_log) and transport ([krabka_client_core]). The public entry point is Controller::start. It spawns the engine and opens a TCP listener. That listener serves the real KIP-595 RPCs (Fetch=1, Vote=52, BeginQuorumEpoch=53, EndQuorumEpoch=54) and the Krabka-private observer and forward RPCs. Controller::start returns a ControllerHandle, which submits metadata changes and reads the current krabka_metadata::MetadataImage.

§Quick start

use {
    krabka_metadata::{MetadataRecord, TopicRecord},
    krabka_raft::{Controller, ControllerConfig, NodeId},
    std::time::Duration,
    uuid::Uuid,
};

let dir = tempfile::tempdir()?;
let cfg = ControllerConfig::for_tests(NodeId(1), dir.path().to_path_buf());
let controller = Controller::start(cfg).await?;

controller
    .submit_change(vec![MetadataRecord::V1Topic(TopicRecord {
        name: "my-topic".into(),
        topic_id: Uuid::new_v4(),
        partitions: 3,
        replication_factor: 1,
    })])
    .await?;

assert2::assert!(controller.current_image().topic("my-topic").is_some());
controller.shutdown().await;

§Capabilities and boundaries

The controller persists and recovers KRaft metadata records, and it serves and installs KIP-630 snapshots through FetchSnapshot. It also publishes the current metadata image to broker tasks and exposes Krabka-private submit and fetch RPCs for broker and observer integration.

KIP-853-style observer bootstrap and auto-join are wired through the broker and controller configuration. ControllerHandle::add_learner stages a node identity; ControllerHandle::change_membership reconciles a one-voter delta into add_voter or remove_voter. Mixed JVM and Krabka controller quorums are outside this crate’s compatibility target.

Re-exports§

pub use event_queue::EventQueueRecorder;
pub use event_queue::NoEventQueueRecorder;
pub use handshake::AllowAllGrants;
pub use handshake::ClusterGrants;
pub use handshake::ClusterOperation;
pub use handshake::ControllerApiVersions;
pub use handshake::HandshakeStream;
pub use handshake::RaftConnection;
pub use handshake::RaftHandshakeError;
pub use handshake::RaftListenerHandshake;
pub use kraft::MetadataFetchSlice;
pub use kraft::controller::control_batch_image_records;
pub use kraft::controller::decode_committed_value;
pub use kraft::controller::is_kip835_noop;
pub use reconfig::AddVoter;
pub use reconfig::ReconfigOutcome;
pub use reconfig::RemoveVoter;
pub use reconfig::UpdateVoter;

Modules§

event_queue: The controller’s event-queue measurements.
handshake: Pluggable inbound handshake for the controller listener.
kraft: Hand-rolled KRaft consensus engine (KIP-595 and KIP-996).
reconfig: KIP-853 requests submitted to the single-owner Raft engine.
voter_requests: The KIP-853 request checks that AddRaftVoter, RemoveRaftVoter and UpdateRaftVoter share on the controller listener and on the broker listener, with Kafka’s codes, order and messages.

Structs§

AcceptingListener: The listener an accept arrived on.
AppData: What the controller asks Raft to replicate.
AppDataResponse
ConnectionGuard: RAII release for one accepted connection.
ConnectionLimiter: Live-connection accounting for one node’s listeners. A clone shares the same counters, so every accept loop and every ConnectionGuard account against one set of totals.
ConnectionLimits: The two caps a node is started with. usize::MAX means unlimited.
ConnectionQuotaSettings: The quotas in force for one listener, before the hosts of max.connections.per.ip.overrides are resolved.
ConnectionQuotas: The quotas one accept is checked against: the settings in force and the resolved override addresses.
Controller: Zero-sized factory for ControllerHandles.
ControllerAdminRequest: Authenticated request handed from the controller listener to the broker’s existing Admin handler registry.
ControllerAdminResponse: Encoded Kafka response body plus its response-header shape.
ControllerApiVersion: One Kafka API version range served by a controller-listener Admin router.
ControllerConfig
ControllerFetchMissLimit: The number of consecutive fetch misses a controller tolerates.
ControllerHandle: Handle returned by Controller::start.
KrabkaMetadataFetchRequest
KrabkaMetadataFetchResponse
KrabkaSubmitChangeRequest: Forward-to-leader payload.
KrabkaSubmitChangeResponse
ListenerLimits: The limits Kafka’s SocketServer puts on every listener, including the controller’s.
MetadataLogConfig: How the metadata log rolls its segments and when it gives up the prefix that a snapshot covers. The defaults are Kafka’s.
MetadataRaftCommandQueueCapacity: The capacity of the metadata Raft command queue.
MetadataRaftFetchMax: The most bytes one metadata Raft fetch carries, a whole byte count that fits the protocol’s int32 field.
Node: KIP-853 voter node identity used by controller membership.
NodeId: A broker, node, replica, or controller identifier.
ObservedStream: A stream that reads and writes through inner and shows observer every poll.
OffsetReservation
PlaintextDialer: Default no-op dialer: it opens a raw TcpStream with Connection::connect.
QuorumState: Krabka-native view of the controller’s current quorum state.
QuorumStateSnapshot: A structured, node-local snapshot of consensus state for the handle, which serves the broker’s DescribeQuorum admin view.
ReleasedApi: One Kafka 4.3.1 request schema: its api key and the versions the release accepts by default.
SnapshotSlice: A contiguous byte window of the latest metadata .checkpoint, returned by ControllerHandle::read_snapshot_range.
SubmitChangeResult

Enums§

BootstrapMode: Bootstrap orchestration for a freshly-formatted controller node.
ConnectionLimit: The connection ceiling that refused an inbound connection.
DelegationTokenMutation: A token mutation bound to the exact committed record observed by its caller.
MetadataReplayError: A committed metadata log record that a controller cannot replay.
PersistedFormatError: What is wrong with a node-local file whose layout carries a version marker. RaftError::PersistedFormat names the file.
RaftError
SnapshotRange: Outcome of ControllerHandle::read_snapshot_range.
UnstableApiVersions: Whether a listener advertises and accepts what Kafka trunk serves beyond the latest Kafka release.
UnstableFeatureVersions: Whether a node supports feature levels past the latest production ones.

Constants§

API_KEY_DELEGATION_TOKEN_MUTATION: Generation-bound delegation-token mutation forwarded to the leader.
API_KEY_METADATA_FETCH: Observer metadata fetch.
API_KEY_SUBMIT_CHANGE: Forwards a Controller::submit_change from a follower to the leader.
DEFAULT_METADATA_LOG_SEGMENT_ROLL_INTERVAL: metadata.log.segment.ms default: seven days.
DEFAULT_METADATA_LOG_SEGMENT_SIZE: metadata.log.segment.bytes default: 1 GiB.
DEFAULT_METADATA_MAX_IDLE_INTERVAL: metadata.max.idle.interval.ms default: 500 ms.
DEFAULT_METADATA_MAX_RETENTION: metadata.max.retention.ms default: seven days.
DEFAULT_METADATA_MAX_RETENTION_SIZE: metadata.max.retention.bytes default: 100 MiB.
KAFKA_4_3_1_APIS: Every api key Kafka 4.3.1 has a valid version of, in key order.
LATEST_PRODUCTION_METADATA_VERSION: Kafka 4.3’s MetadataVersion.LATEST_PRODUCTION, 4.3-IV0.
MAX_CONNECTIONS: Kafka’s SocketServerConfigs.MAX_CONNECTIONS_CONFIG.
MAX_CONNECTIONS_PER_IP: Kafka’s SocketServerConfigs.MAX_CONNECTIONS_PER_IP_CONFIG.
MAX_CONNECTIONS_PER_IP_OVERRIDES: Kafka’s SocketServerConfigs.MAX_CONNECTIONS_PER_IP_OVERRIDES_CONFIG.
MAX_CONNECTION_CREATION_RATE: Kafka’s SocketServerConfigs.MAX_CONNECTION_CREATION_RATE_CONFIG.
METADATA_PARTITION_DIR: The name of the metadata partition’s directory under the metadata log directory: Kafka’s UnifiedLog.logDirName of __cluster_metadata-0.
MIN_METADATA_LOG_SEGMENT_SIZE: The least metadata.log.segment.bytes Kafka accepts: 8 MiB. Kafka’s ConfigDef refuses a smaller value, and only its internal internal.metadata.log.segment.bytes goes below it, for tests.
PRIVATE_BASELINE_VERSION: The version of every krabka-private API that a sender uses when it has no metadata image to read krabka.version from: v0, the krabka-broker 1.0.0 formats, which every 1.x receiver serves.
PRIVATE_CLUSTER_AUTHORIZATION_FAILED: The error_code of a krabka-private response when the connection principal lacks ClusterAction on the cluster. It is Kafka’s CLUSTER_AUTHORIZATION_FAILED.
PRIVATE_UNSUPPORTED_VERSION: The error_code of a krabka-private response to a request at a version the receiver does not implement. It is Kafka’s UNSUPPORTED_VERSION.
SUBMIT_CHANGE_UNCOMMITTED_TAIL: KrabkaSubmitChangeResponse::error_code: the leader refused a compare-and-set until its uncommitted tail commits. The forwarding node turns it back into crate::RaftError::UncommittedTail.

Traits§

ControllerAdminRouter: Optional KIP-919 Admin RPC surface attached by the broker crate.
OutboundDialer: Outbound dialer the controller hands to the peer sender.
RaftShardRouter: Classifies and serves shard-addressed KIP-595 requests before metadata dispatch.
RegistrationAdmission: What the broker crate does to a batch that registers a broker anew, before the leader appends it.
StreamObserver: What an ObservedStream reports to.

Functions§

api_versions_max_version: Highest ApiVersions version a listener serves under unstable.
describe_quorum: Answers a decoded DescribeQuorum request from quorum, this node’s consensus snapshot.
deserialize_metadata_snapshot: Decode the KIP-630 metadata records from a Kafka metadata snapshot.
deserialize_metadata_snapshot_image: Decode the whole image a Kafka metadata snapshot holds: its KIP-853 quorum controls, as a V1KRaftVersion and a V1Voters record, ahead of its KIP-630 metadata records.
fetch_error_response: Kafka’s FetchRequest.getErrorResponse: error_code and the session id of request at the top level.
finalized_feature_keys: The finalized-feature rows of image, in name order: every finalized level above 0, metadata.version at Kafka’s minimum before the first feature delta, and kraft.version when its level is above 0.
is_valid_api_versions_request: Kafka’s ApiVersionsRequest.isValid.
is_valid_client_info: KIP-511 client-information validity check. Matches the JVM ApiVersionsRequest.isValid regex [a-zA-Z0-9](?:[a-zA-Z0-9\-.]*[a-zA-Z0-9])?:
kafka_4_3_1_api: The Kafka 4.3.1 row for api_key, or None when that release has no such api key.
kafka_4_3_1_max: The highest version of api_key Kafka 4.3.1 accepts by default, or None when that release has no such api key.
metadata_log_nonempty: True when the metadata partition directory dir already holds durable raft state, which a node that ran before leaves. A quorum-state file or a log segment that holds bytes is such state.
metadata_partition_dir: The metadata partition directory under metadata_log_dir, Kafka’s metadata.log.dir. It holds the metadata log segments, the KIP-630 <end_offset>-<epoch>.checkpoint snapshots and the quorum-state file.
metadata_snapshot_last_contained_log_timestamp: Read the append timestamp of the last record batch contained in a KIP-630 metadata snapshot.
private_api_highest_version: The highest version of the krabka-private api_key this build serves, or None when api_key is not one of them.
private_request_version: The version of rpc a sender puts in the request header, negotiated from image, the sender’s current metadata image.
serialize_bootstrap_snapshot: Serialize the bootstrap checkpoint of a dynamic format, as Kafka’s Formatter.writeBoostrapSnapshot writes it: the KIP-853 kraft.version and voter set, then the bootstrap records in their own order.
serialize_metadata_snapshot: Serialize a Kafka metadata snapshot, including KIP-853 control state.
supported_feature_key: The supported-feature row for one feature at api_version, or None when Kafka leaves the feature out of the response.
supported_feature_keys: Every supported-feature row of the krabka_metadata feature registry at api_version, each feature capped as unstable caps it (crate::supported_feature_range).
supported_feature_range: The supported range of feature under unstable.
supported_feature_ranges: The feature ranges a node registers with the controller under unstable.
unsupported_version_response: Kafka’s ApiVersionsRequest.getErrorResponse for UNSUPPORTED_VERSION (KIP-511).

Type Aliases§

ControllerAdminRouteFuture
IpOverrides: The cap of each override address.
RegistrationAdmissionFuture: The batch RegistrationAdmission::admit answers with.
ShardRouteFuture: Optional router for KIP-595 traffic addressed to non-metadata quorum shards.