Release candidate qualification

This record is the contract for the next promoted release candidate. It gives the procedure that freezes a candidate, the gates that must pass on that exact candidate, and the proposed list of release blockers. Issue #1401 owns it.

This record does not authorize a release. It does not tag, sign, publish or promote anything. Releasing is the release procedure, and it stays unchanged: a candidate that passes every gate here is released by that procedure, from a vX.Y.Z tag on main.

Status, 2026-10-25. v1.0.6, v1.0.7 and v1.0.8 are released, and all three contain #1540. All three pass aspect verify-release with no problems, so the condition of blocker B1 is met. The proposed next ecosystem candidate is v1.0.8 at 3831a25d, tag object b4f543b4, with the amd64 delivery image sha256:960d3ad1…. Its protocol dependency is 0.7.0 at 1b282837, its Rust client 0.7.1 at ba3443d0, and its schema registry 0.4.3 at ac4fe56e. The draft manifest ecosystem-v1.0.8.json and the proposed freeze record candidate-freeze-3831a25d-proposed.json wait for maintainer review. Freezing is the maintainer’s decision. The CLI, operator and website components of the draft still carry their baseline values, and the review must set them. Until a candidate is frozen, no run here is candidate evidence. The remaining qualification runs are scheduled with the 1.1 release qualification.

The eight-gate preflight exposed failures in single-node transaction configuration, CLI admin response handling and topic configuration, audit placement during operator evacuation, and the schema adapter’s test ownership. Those fixes are merged in the website, CLI and broker. Broker #1573 also moves feature validation before recovery writes, so the loaded lifecycle uses the startup-safe v1.0.6 baseline. Every gate must run again on the final candidate; preflight passes do not qualify v1.0.8.

The cloud qualification workflow provides the performance and 72-hour soak workers. Independent pilot acceptance still requires the named human operators and results in the pilot plans.

Release and ecosystem claims

Krabka makes two separate claims, as Releasing and Ecosystem qualification say:

• A broker release is a signed image and a GitHub release for a commit whose main CI run passed. This claim alone does not qualify an ecosystem.
• An ecosystem-qualified stack is a broker release whose frozen candidate passed every gate in Candidate gates, with the sibling set that the candidate manifest names.

Describe a stack as ecosystem-qualified only when every gate below passed on the frozen candidate and every agreed release blocker is closed. A partial pass, an older run, or a run on another revision is not enough. On 2026-10-25 no 1.x release is ecosystem-qualified. The last qualified stack is krabka-broker v0.6.1 with the sibling set in ecosystem-eight-gate-baseline.json.

Freeze a candidate

A candidate is three things that must not move while it is qualified:

1. The krabka-broker commit. It is on main, and its ci.yml push run passed gate.
2. Every sibling revision that the commit’s [patch.crates-io] block pins. These revisions are compiled into the image.
3. The amd64 image digest that delivery recorded for the commit, in the delivery-image artifact of that run. The artifact expires after 90 days.

Do these steps in sequence:

1. Pick the commit. Its ci.yml push run must be success. Get that run’s delivery-image artifact:

   run="$(gh api "repos/krabka-io/krabka-broker/actions/workflows/ci.yml/runs?head_sha=${COMMIT}&event=push" \
     --jq '[.workflow_runs[] | select(.conclusion == "success")][0].id')"
   artifact="$(gh api "repos/krabka-io/krabka-broker/actions/runs/${run}/artifacts" \
     --jq '.artifacts[] | select(.name == "delivery-image") | .id')"
   gh api "repos/krabka-io/krabka-broker/actions/artifacts/${artifact}/zip" >delivery-image.zip
   unzip -o delivery-image.zip

2. Freeze the commit, the pins and the digest. The task checks the digest against the registry, and lists each component of the manifest whose revision is not the pin that the image compiles in:

   aspect freeze-candidate --commit "${COMMIT}" \
     --delivery-digest delivery-image.digest \
     --manifest qualification/ecosystem-eight-gate-baseline.json \
     --output candidate-freeze.json

3. When the task prints a DRIFT line, write a reviewed candidate manifest. Start it with aspect check-qualification --new-candidate, and set every component to its pinned revision and to the artifacts built from that revision. The workflow’s broker_revision and broker_digest overrides replace the broker component only. With drift, they describe siblings that the image was not built from.

4. Commit the candidate manifest and the freeze record under qualification/ in one pull request. From that merge on, every gate below runs against these values and no others.

5. Rebuild nothing. A fix to a blocker makes a new commit, and a new commit is a new candidate. Start again at step 1, and run every gate again.

The dry run of these steps on 948e3cc8 is in candidate-freeze-948e3cc8-dry-run.json. It is not a candidate. It was written by the schema 1 task, which compared every [patch.crates-io] pin with the manifest, so it lists all ten pinned siblings as drift. Seven of them, krabka-gateway among them, are patches that nothing in the dependency graph uses: Cargo.lock keeps them under [[patch.unused]], and the image does not contain them. The task now records them as unused_patches and compares only the siblings that the image compiles in. Those are krabka-protocol, krabka-client-rs and krabka-schema-registry, and all three still differ from the qualified baseline manifest. So the next qualification needs a reviewed manifest that names those three revisions; the broker overrides alone are not correct.

Candidate gates

Each gate must pass on the frozen candidate. “Needs” names what the run needs that a pull-request runner does not have. The status column is the state on main on 2026-10-18, and it is not candidate evidence.

Since 2026-10-18. The status column above is not current. These runs changed it, and none is candidate evidence:

• Required CI passed gate at v1.0.8 in push CI 39518736364. On main at 315c2985, push CI 40160834355 failed gate: coverage failed feature_finalization_test::validate_only_does_not_persist and dynamic_voters_test::remove_voter_shrinks_quorum.
• The full container matrix, GSSAPI, the upgrade lane and the four-hour soak passed at v1.0.8 in dispatch 39519220181, and at v1.0.7 in dispatch 39493788635.
• The criterion regression lane failed in both dispatches on a measured verdict: at v1.0.8, broker/produce_acks/32part/acks_all at 1.16 times the v1.0.5 reference, against a 0.113 tolerance; at v1.0.7, log/read/past_end_returns_empty at 1.79 times. In the nightlies of 10-22 to 10-24 it failed in its harness, because the cargo log named no bench executable, so no comparison ran.
• The regression budgets, the eight gates, recovery and lifecycle are in B8, B5 and B10. Provenance is green for v1.0.6 to v1.0.8.

Rules for the gates:

• Run every gate on the frozen commit and image digest. A gate that ran on another commit gives no evidence for the candidate.
• Keep every failed run. A failure is a blocker until a later run on a new candidate passes, or the maintainer agrees that the item is a known limitation.
• Re-run a failed job once only to show an infrastructure failure, such as the image pull failures of run 39114726914. A second failure is a real failure.
• Do not skip, disable or ignore a test to make a gate pass.

Provenance verification

release.yml does these steps for a vX.Y.Z tag:

1. It gets the amd64 digest from the delivery-image artifact of the successful main push run, and checks it against the registry with aspect check-delivery-image.
2. It builds and pushes the arm64 manifest, and joins both children by digest into the vX.Y.Z index.
3. It signs the index with cosign sign, and attests the CycloneDX SBOM with cosign attest --type cyclonedx and the provenance with cosign attest --type slsaprovenance. The identity is the keyless GitHub OIDC identity of release.yml at the ref that ran it.
4. It runs cosign verify and cosign verify-attestation against the index, and uploads sbom.cdx.json to the GitHub release.

aspect verify-release repeats these checks from outside, against a release that exists. It also checks the content of the attestations, and pins the support documents at the tagged commit:

git fetch origin tag v1.0.5
aspect verify-release --tag v1.0.5 \
  --delivery-digest delivery-image.digest \
  --output release-verification
# A release that a workflow_dispatch from main re-ran:
aspect verify-release --tag v1.0.0 --identity-ref refs/heads/main
# The support documents must be the bytes an earlier record pinned:
aspect verify-release --tag vX.Y.Z --expected-support candidate/release-verification.json

The task fails when any of these is false:

• The index is signed by release.yml under the identity of the ref that ran it.
• The index has one amd64 child and one arm64 child.
• The provenance statement is SLSA v1, its subject is the index, and its predicate names the tagged commit, the tag ref, the arm64 child, and the amd64 child as the tested delivery image.
• The SBOM attestation’s subject is the index, and its predicate is the release’s sbom.cdx.json asset, compared as JSON.
• The amd64 child is the digest of the delivery-image artifact, when --delivery-digest is given.
• Each support document exists at the tagged commit, and matches the record that --expected-support names, when one is given.

The support documents are README.md, CHANGELOG.md, SECURITY.md, docs/persisted_formats.md, docs/format-divergences.md, docs/KIP_MATRIX.md, docs/config-reference.md, docs/releasing.md, docs/qualification.md, and every file under docs/operations/. The record gives the SHA-256 of each one. A page added under docs/operations/ later, such as the #1400 support matrix, is pinned with no change to the task. To bind the documents to the delivered bytes, keep the candidate’s record, and check the release against it with --expected-support. Because the tag names the commit and the provenance names the same commit and the delivered digests, a pass ties the documents to the image bytes.

Use cosign 3. cosign 2 finds no signature on these releases, because sigstore/cosign-installer v4 signs with cosign 3, which stores the signature as an OCI referrer.

Result for v1.0.5

The task ran on 2026-10-18 against v1.0.5 and v1.0.0. The record for v1.0.5 is release-verification-v1.0.5.json.

v1.0.0, which a dispatch from main released, gives the same result under --identity-ref refs/heads/main.

The provenance defect is in release.yml. It writes a SLSA v1 predicate (buildDefinition and runDetails) but attests it with --type slsaprovenance. cosign parses that type into the SLSA v0.2 schema, so every field is dropped, and the statement carries no commit, no ref and no image source. cosign verify-attestation --type slsaprovenance still passes, because it checks the signature and the type, not the content. A local cosign attest-blob test with cosign 3.0.2 shows that the type https://slsa.dev/provenance/v1 keeps the predicate as written. The fix is blocker B1 below, in #1509. It does not change the attestations of v1.0.0 to v1.0.5, so those releases keep an empty provenance predicate. For them, the release run log and the delivery-image artifact are the provenance evidence.

Result for v1.0.6, v1.0.7 and v1.0.8

The three releases after #1509 pass. The task wrote release-verification-v1.0.6.json on 2026-10-20, and release-verification-v1.0.7.json and release-verification-v1.0.8.json on 2026-10-25 with cosign 3.1.4. Each record has "verified": true and an empty problems list. The v1.0.7 and v1.0.8 records were run with --delivery-digest.

All three tags are on main, and all three contain #1540, the diskless restore fix.

Blocker list (proposal)

The maintainer agrees this list on #1401. Each item has one class:

• Release blocker: the candidate cannot be promoted, or described as ecosystem-qualified, until the item is fixed and the fix is verified on a new candidate.
• Known limitation: the release can ship with it, and the support documents (#1400) must state it.
• Not relevant: the item does not affect the candidate.

Release blockers

F1 to F4 are blockers to resolve, not documented flakes, because each one is in a lane that gate requires. F5 is in a lane that gate does not require, but it checks on-disk safety during a rollback. A fix makes the test deterministic. It does not skip the test or add a retry.

Known limitations

Not relevant