Tiered storage
krabka-remote-storage and krabka-object-store upload sealed log segments to S3-compatible object storage (including MinIO) or Google Cloud Storage. Active segments stay on local disk, and consumers can fetch older offsets from object storage.
Two-tier storage architecture
Partition segments use two storage tiers:
Hot tier (local disk)
Active write-ahead log (WAL) segments are appended and indexed on local disks via krabka-log, in Apache Kafka's on-disk log format.
Cold tier (object storage)
Once a segment reaches its configured size or time threshold, background copier threads upload it to an S3-compatible object store (Amazon S3 or MinIO) or Google Cloud Storage via krabka-remote-storage.
Topic-backed remote log metadata manager
krabka-remote-storage-topic stores the segment catalog in the internal Kafka topic __remote_log_metadata. Kafka replicates that topic; the catalog does not require a separate database.
# broker.toml - KIP-405 tiered storage on an S3-compatible store
# Use exactly one of storage_dir, [remote_storage.s3] or [remote_storage.gcs].
[remote_storage]
copier_threads = 10
reader_threads = 10
index_cache_size = "1GiB" # on-disk cache of remote segment indexes
[remote_storage.s3]
bucket = "krabka-data-lake-archive"
region = "us-west-2"
prefix = "cluster-a"
# endpoint = "http://minio:9000" # MinIO or another S3-compatible store
# allow_http = true # only for an endpoint without TLS
# The topic-backed remote log metadata manager (__remote_log_metadata)
[remote_storage.kafka_metadata]
num_partitions = 50
replication = 3Remote fetch routing
Consumers fetching offsets that have aged out of local storage are served from object storage through a bounded pool of remote reader threads, using an on-disk cache of remote segment indexes. Tokio async tasks handle socket transfer.