Struct S3RemoteStorage 

pub struct S3RemoteStorage { /* private fields */ }

A RemoteStorageManager backed by any S3-compatible object store.

Construct it with S3RemoteStorage::with_store, which accepts any ObjectStore impl, for in-process tests. Use S3RemoteStorage::from_s3_config for the production path, which builds an AmazonS3 client from credentials, endpoint, and bucket.

Implementations

impl S3RemoteStorage

pub fn from_gcs_config(cfg: &GcsConfig) -> Result<Self, RemoteStorageError>

Builds a GoogleCloudStorage client from cfg and wraps it in the generic S3RemoteStorage engine.

With no credential fields set, authentication uses Workload Identity or ADC through the metadata server. This is the keyless GKE path.

Errors

Returns RemoteStorageError::InvalidArgument if object_store’s builder rejects the bucket, credential, and endpoint combination. For example, the caller supplied both a service-account path and a key, a credential file is unreadable, or the bucket name is empty.

impl S3RemoteStorage

pub const DEFAULT_MAX_INDEX_BYTES: u64

Largest index read by default: Kafka’s Java ByteBuffer capacity fits in a signed int, so an index larger than this cannot be loaded by Kafka. The cap applies before buffering and during streaming.

pub fn with_max_index_bytes(self, max_bytes: u64) -> Self

Set a lower memory limit for whole index reads, including snapshots and epoch checkpoints. Values above Self::DEFAULT_MAX_INDEX_BYTES use that ceiling. An object above the limit returns an error; it is never truncated. Zero permits only empty objects.

pub fn with_store(store: Arc<dyn ObjectStore>, prefix: Option<String>) -> Self

Wraps an arbitrary ObjectStore, for example object_store::memory::InMemory for tests. Use Self::from_s3_config for the production S3 path. Multipart tuning falls back to the DEFAULT_MULTIPART_THRESHOLD and DEFAULT_MULTIPART_CHUNK_SIZE constants. Call Self::with_multipart_tuning to override them in tests.

pub fn with_worm(self, cfg: &WormConfig) -> Result<Self, RemoteStorageError>

Puts this backend into WORM archive mode.

Every copy then seals a signed, chained .manifest beside the segment, every delete is refused, and a write_only archive refuses remote fetches as well.

Errors

Returns an error when the bucket policy cannot be confirmed or cfg’s signing key cannot be loaded.

pub fn with_multipart_tuning( self, threshold: ByteSize, chunk_size: ByteSize, ) -> Self

Overrides the multipart threshold and chunk size. Returns self for chained calls. Tests use this to force the multipart path on small fixtures. Production usually keeps the defaults.

pub fn from_s3_config(cfg: &S3Config) -> Result<Self, RemoteStorageError>

Builds an AmazonS3 client from cfg and wraps it.

Errors

Returns RemoteStorageError::InvalidArgument if object_store’s builder rejects the bucket, region, and endpoint combination.

Trait Implementations

impl Debug for S3RemoteStorage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl RemoteStorageManager for S3RemoteStorage

fn copy_log_segment_data( &self, metadata: &RemoteLogSegmentMetadata, data: &LogSegmentData, ) -> Result<Option<CustomMetadata>, RemoteStorageError>

Copies a segment’s data and all of its indexes to the remote tier.

fn fetch_log_segment( &self, metadata: &RemoteLogSegmentMetadata, start_position: u32, end_position: Option<u32>, ) -> Result<Vec<u8>, RemoteStorageError>

Fetches a byte range of a segment’s .log data.

fn fetch_index( &self, metadata: &RemoteLogSegmentMetadata, index_type: IndexType, ) -> Result<Vec<u8>, RemoteStorageError>

Fetches one of a segment’s indexes in full.

fn delete_log_segment_data( &self, metadata: &RemoteLogSegmentMetadata, ) -> Result<(), RemoteStorageError>

Deletes a segment’s data and all of its indexes from the remote tier.