Skip to content

Integration suite: zero-red + nightly + a real --full gate (integration-suite-gate)

2026-08-03

The stack-backed @numero/test suite had decayed to 146 red files — not one bug but five compounding classes (ambient-env drift, chain/permit seam rot, stale tests against retired models, readiness races, cross-file env pollution). Nothing ran it on a schedule, so pnpm release --full was a gate in name only.

Now:

  • Zero-red is the invariant. The arc ran 146 → 79 → 72 → 50 → 19 → the final stragglers closed (order-interaction leaks, scaffolded phantoms, retired premises, a real two-clock betting-window rewrite). Failure SETS, not counts, were compared at every step — zero new red across the drive.
  • Every disabled test says why. check:skip-reasons (chained into pnpm typecheck) fails the build on any bare .skip; all 99 sites carry a greppable SKIP-REASON(<token>) from a small taxonomy (cage-only is crypto-at-the-cage's deletion bill; retired-premise, unbuilt-harness, solana-grant-leg, …). See the integration-suite guide.
  • Nightly CI. .github/workflows/integration-suite.yml runs the suite against staging HEAD every night on a hosted runner (trimmed 16-service roster; pinned+cached Agave; cached anchor artifacts), uploads the log + failure diagnostics, and emits one ci.integration_suite.completed span. Dash0 rule 13 alerts Slack on ERROR; a separate absence watchdog emits the ERROR span itself if no run completed in 26h (the rule-6 inexpressible-absence class). Runbook: integration-suite-red.
  • pnpm release --full is real. A preflight validates the suite's environment (docker, the worktree's test stack, the validator via the one declared resolver) and refuses with a single named reason before anything moves — never a mid-run abort. See releasing.

Surfaced along the way and filed for owners: the lobby live-map ghost entries, the top-up self-heal felt staleness, hosted-agent inference latency under 9-way load (shared dev Gemini key), and the solana-cage betting-leg failure set (now diagnosable — the artifacts prereq and the master kill-switch fixture are solved).