Ecosystem qualification

qualification/ecosystem-eight-gate-baseline.json is the qualified baseline: krabka-broker v0.6.1 with krabka-operator v0.4.1 and the rest of its sibling set, which passed all eight gates in qualification-36293960620. It is the default manifest for the workflow and for aspect check-qualification. Schema 2 binds each executed check to a candidate and the exact qualification adapters. It requires the disaster-recovery gate owned by issue 555 and the schema-evolution, registry-migration, and snapshot-retention gates owned by issues 556-558.

qualification/original-four-gate-baseline.json is the completed historical schema-1 candidate for the original four gates. Keep its revisions, evidence and report intact.

Run a subsequent candidate

The ecosystem qualification workflow accepts a repository-relative manifest and an execute switch. With execute=false, it verifies the existing published result and its hashes. With execute=true, it creates a fresh draft from that component set, runs eight independent jobs, and publishes a qualification release only after every job succeeds.

Supply both broker_revision and broker_digest to qualify a newly delivered broker using the same sibling set. The prepare job applies those overrides, freezes the complete input, verifies revisions against each recorded default branch, and distributes that exact input to every job. This needs no candidate commit after the delivery image has been published. To change sibling revisions or artifacts, provide a reviewed manifest containing that full candidate set.

To qualify a new broker delivery against the baseline sibling set, run the baseline with the delivered revision and digest. When a run with a new sibling set publishes, check in its qualified manifest as the next baseline; do not invent a draft manifest or mutable image tag to make a run appear green.

gh workflow run qualification.yml \
  -f manifest=qualification/ecosystem-eight-gate-baseline.json \
  -F execute=true \
  -f broker_revision="$DELIVERED_COMMIT" \
  -f broker_digest="$DELIVERED_IMAGE_DIGEST"

The broker digest must come from the successful delivery run for that commit. The qualification workflow exercises those bytes; release promotion separately verifies the delivery record. Do not infer a delivery digest from a mutable tag.

The matrix has fail-fast: false. GitHub’s Re-run failed jobs retries only failed gates with the original uploaded candidate; successful gates remain available. Re-running the entire workflow creates a new candidate timestamp and reruns every gate. The jobs require Linux amd64, Docker, and sufficient capacity for the existing stack; installation and lifecycle also install pinned Helm and Kind. Their timeouts bound the run rather than turning a timeout into a pass.

What runs

Gate	Executed boundary
Installation	The pinned public Compose and Helm recipes with candidate images and downloaded, checksum-verified charts; produce/readback and uninstall. The existing Go and Java integration suites run against the installed broker, and the pinned schema registry’s RF=3 live-registry suite plus the broker’s incompatible-protocol build probe rerun the original compatibility supporting checks.
Operator lifecycle	The pinned operator `packaging/kind-lifecycle.sh`, using published chart contents and the candidate broker, operator and rebalancer images; upgrade, disruption, scale-down, certificate rotation and acknowledged-record reconciliation.
Authenticated CLI	The pinned CLI’s real `candidate_broker` test, against a disposable three-broker SASL cluster. Its one executable lookup is adapted in the isolated checkout to invoke the downloaded CLI binary instead of a rebuilt binary; the patch is retained.
Observability recovery	The pinned demo Compose recipe with explicit candidate broker, o11y and demo-app image overrides on linux/amd64, followed by its `qualify-failover.sh`; signal queries, WAL recovery, offset reconciliation and alert firing/resolution.
Disaster recovery	A TLS and SASL/SCRAM-SHA-512 RF=3 cluster created from the candidate broker image, a locked MinIO archive, and the image’s bundled backup, restore, and WORM verifier. The job captures offsets and configuration at an explicit boundary, destroys every source data directory, restores a fresh cluster, reconciles records at their original offsets, settings, and consumer positions, and records measured RPO/RTO. Tampered, missing, untrusted-head, invalid-certificate, invalid-credential, and denied-ACL probes must fail without a complete capture. Classic and diskless topics are both required.
Schema evolution	Confluent-derived Avro, JSON Schema, and Protobuf compatibility matrices, serializer reference/cache tests, RF=3 broker validation, and an old/new record round trip across broker and registry restarts.
Registry migration	Full compacted `_schemas` lifecycle replay plus forward and reverse handoff against the pinned Confluent Schema Registry image.
Snapshot retention	Exact retained-epoch diagnostics, bounded reclamation, concurrent restore protection, and process-kill recovery during save and reclaim.

The lifecycle harness expects a local operator tag. Its private alias is checked against the pulled candidate image ID before execution. It packages source charts as ancillary evidence, but installation uses the verified downloaded chart contents copied into the isolated harness tree. No user checkout is modified. Supporting source tests remain distinguishable from installed-image checks in the command log.

A test process exiting successfully is insufficient: adapters check named live test results or behavioral outcomes and record the checks that actually ran. Skipped or zero-test runs cannot produce a valid gate receipt. Historical recovery and CDC evidence remains historical; this workflow does not relabel those old runs as fresh candidate results. The monthly schedule is enabled only after the first schema-2 eight-gate manifest is checked in as the immutable baseline.

Local execution and evidence

aspect check-qualification --new-candidate qualification/candidate.json
# Edit candidate.json to name the intended immutable revisions and artifacts.
aspect check-qualification --manifest qualification/candidate.json
aspect check-qualification --manifest qualification/candidate.json \
  --gate cli-admin --output qualification-run-1

Every gate invocation uses a new temporary checkout. Choose a fresh --output for retries; existing evidence is never silently replaced. Failed attempts keep their command log and scratch directory for diagnosis. Successful receipts name the gate, candidate SHA-256, adapter SHA-256, executed check count and names, command hash, and log hash. The candidate identity covers its timestamp, architecture, all repository revisions/default branches, and artifact references/digests. Reordering components or adding output evidence does not change that identity.

Run all eight gates into the same output directory, then assemble:

aspect check-qualification --manifest qualification/candidate.json \
  --assemble --output qualification-run-1 \
  --publish-prefix https://github.com/krabka-io/krabka-broker/releases/download/qualification-RUN_ID
aspect check-qualification --manifest qualification-run-1/manifest.json --final

Assembly verifies every receipt and retained log/command hash before writing a qualified manifest, report and per-gate bundles. It rejects another candidate’s receipt, missing gates, failed commands and zero executed checks. Local assembly does not publish anything. The workflow uploads the assembled artifacts to a draft qualification-RUN_ID release, downloads them to verify checksums, then publishes it as a prerelease without changing latest. Published qualification releases are never overwritten by retries.