Tiered Storage with NVMe Caching
Decouple broker compute from storage capacity (krabka-remote-storage and krabka-object-store). Store petabytes of historical streaming data cost-effectively in S3, GCS, or MinIO while serving real-time consumer fetches from NVMe flash.
Two-Tier Storage Architecture
Krabka splits partition lifecycle management across two high-performance storage tiers:
Hot Tier (Local NVMe Direct I/O)
Active write-ahead log (WAL) segments are appended and indexed directly on local NVMe SSDs via krabka-log, optimizing for microsecond-level produce acknowledgments.
Cold Tier (Object Storage Offloading)
Once a segment reaches size or time limits, background workers compute CRC32C checksums and stream the segment into Amazon S3, Google Cloud Storage, or MinIO via krabka-remote-storage.
Topic-Backed Remote Log Metadata Manager
Segment catalog state is tracked durably in the internal compacted topic __remote_log_metadata (krabka-remote-storage-topic). This design guarantees that metadata lifecycle, replication, and failovers are managed natively by the KRaft quorum without requiring external relational databases or DynamoDB tables.
Tiered Storage Configuration
[storage.tiered]
enabled = true
backend = "s3" # s3 | gcs | azure | minio
bucket = "s3://krabka-data-lake-archive"
region = "us-west-2"
local_cache_size_bytes = 107374182400 # 100 GiB local NVMe cache
upload_concurrency = 8
segment_upload_timeout_secs = 60Zero-Copy Cache & Direct Fetch Routing
Consumers fetching historical partition offsets are automatically routed through the local NVMe cache or streamed directly from object storage via byte-range requests. Tokio async tasks handle socket transfer with zero JVM heap buffer allocations.