Skip to main content

Module offsets

Module offsets 

Source
Expand description

Committed consumer-group offsets: the one restore input that is not a file on a node.

__consumer_offsets is compacted and internal, so it is never tiered and a KIP-405 archive does not hold it. A restore therefore rebuilds the log and nothing that points into it, and every group starts again from auto.offset.reset. What can be copied is the coordinator’s answer: the committed offset of every group, read with ListGroups and OffsetFetch while the cluster is up, and written back with OffsetCommit once a restored cluster is.

The write-back is Kafka’s Admin.alterConsumerGroupOffsets, the call kafka-consumer-groups --reset-offsets --execute makes: a simple-consumer commit, with an empty member_id and generation -1, that the coordinator accepts for a group with no live members. A group with live members is fenced, which is the correct answer: an offset must not move under a consumer that is reading from it. The admin client retries the coordinator codes Kafka’s AlterConsumerGroupOffsetsHandler retries, so a freshly booted broker that is still creating __consumer_offsets is waited for rather than reported as a refusal.

Structs§

CommittedOffset
One group’s committed offset in one partition.
GroupOffsets
One group, with every partition it has a committed offset for.
GroupOffsetsFile
The captured offsets of every group in one cluster.

Constants§

GROUP_OFFSETS_VERSION
Version of group-offsets.json, its required top-level "version" field.

Functions§

admin_offsets
One group’s captured offsets, keyed the way the admin client commits them.
commit_outcome
How many offsets one group’s commit put back, or why it put back none.
commit_refusals
Every partition the coordinator refused, as topic-partition: NAME (code).
group_offsets
Turn one OffsetFetch answer into the captured shape.