Incident write-up · June to August 2026
Every check green, no data
For June 2026 the state’s food-inspection feed delivered 2,728 observations against a trailing six-month median of 12,362. Every run finished, every check passed, and nothing went red. The hole surfaced two months later during a manual completeness decomposition. By then the state’s rolling extract had moved on, and twenty days of June were gone for good. It was one of several silent failures with the same shape: a job that completes is not a job that delivers.
- Impact
- June inspections at 22% of normal; twenty days lost permanently
- Time to detect
- About two months
- Found by
- A manual completeness check, not an alert
- Root cause
- Every gate checked structure; none checked volume
Written from the build record. Collection has been off since early September 2026.
The hole
Timeline
A ship-ready sign-off finds three weeks of stale data. Its root cause: “all schedules manually disabled.”
The daily state-licensing jobs stop.
A live-health audit finds 33 of 64 monitored source contracts stale and zero active schedule triggers. All 20 checked-in ingest workflows are manual-only. One registry feed shows “65 successful ledger runs but zero observations.” The audit notes that “the existing automated audits themselves emitted no P0 finding.”
The repository variable that gates every scheduled job “sat at false for six days and nothing said anything.” Alerting sat behind the same gate, and the watchdog had no schedule of its own.
The full food-license ingest stops completing. Its log goes quiet; see the stall below.
Observers are ungated, and the watchdog starts running every 6 hours. Its first scheduled tick, at 18:04 UTC, fails with a 403 reading the gate; fixed the same day.
The June hole is named in a commit. A volume-anomaly detector, the bind-parameter fix, self-expiring ledger rows, phase timing logs and a raw-byte archive land the same day.
Root cause
In the commit that named it: “Every existing gate is structural: did the run finish, did shard arity hold, did identifiers stay unique. None notices a clean run over a truncated file.”
The schedule gate made it worse. As of August 10, alerting sat behind the same gate it should have been watching, and the watchdog itself only ran when dispatched by hand. A run skipped by the gate still reports as completed, so the dashboard showed a heartbeat while nothing ran.
And the loss was permanent: “Florida’s extracts are rolling windows, not archives. The DBPR inspection feed publishes about two months.” The twenty days lost to the June outage could not be fetched again.
A related stall
A full food-license ingest resolved about 68,000 entities and passed every ID in one IN list: about 68,000 bind parameters on a single statement, over the Postgres limit of 65,535. The log went quiet, and the process died more than 30 minutes later when the gateway dropped its advisory lock. The feed had not completed since August 11. The fix pages the list at 2,000.
Detection
Not by an alert. The June hole surfaced during a manual completeness decomposition, two months after the fact. The August 10 audit that found the stale contracts was also a manual pass. That is the finding: the system had no automated way to notice a quiet failure.
Fix
Watchdog ungated, August 16
Observers no longer sit behind the gate they police. The watchdog runs every 6 hours, measures the last successful run rather than the last completed one, watches the ingest lanes, and fails loudly when the global schedule gate is off. From its header: “a paused platform must be visible, never quiet.”
Volume-anomaly detector, August 17
Rates each scheduled source’s newest complete month against the median of the six before it: below 50% is critical, below 75% a warning. Its first version flagged a delta feed as a “96% collapse”; delta feeds now go unrated.
Raw-byte archive, August 17
Content-addressed copies of every fetched file, so the next outage cannot erase history. Built; it needed storage credentials and was not confirmed live before collection stopped.
Operations fixes, August 17
Bind parameters paged at 2,000. Silent phases log their duration. A failed ingest expires its own ledger rows, after stuck rows had blocked two feeds for hours and needed repair by hand three times.
Verification
- The detector is tested on the real history. Its test fixture is the inspection feed’s actual monthly series, June hole included.
- A coverage gauge, separate from freshness. Its first reading: 42,221 of 66,097 state-claimed inspections held, or 63.9%. A re-link repair for 46,810 misrouted observations was committed in dry-run mode and not yet run.
- Report-only. The detector was wired onto the freshness lane as report-only and had not been promoted to fail on a critical finding before collection stopped.
How it ended
Collection stopped in early September, and I closed the line on September 27.
What changed after
- Freshness and coverage are separate gates. A feed can be fresh and still be missing most of its rows.
- Heartbeats are measured by success. A watchdog must never sit behind the gate it polices.
- Keep the raw bytes. A state feed is a rolling window, and a gap that outlives it is permanent.
The lessonBuilt is not running. A job counts once it runs on a schedule and something checks that it delivered.