← All decisions

#0057 · 2026-08-09 · accepted · architecture

Fix the Breakout stable-channel deploy crash and replace decision 0046's manual-only stable promotion with CI-gated automatic promotion

deploymentwebsiterelease-channelcibreakout

Context

glemy-website's deploy.yml started crashing on every run after decision 0053's Breakout catalog integration landed (two real failed CI runs): its new Breakout copy step unconditionally ran `cp glemy-stable/breakout.html ...`, but glemy-stable is checked out at whatever commit STABLE_GLEMY_REF names, and that pin (decision 0046) predates Breakout's existence entirely -- the file genuinely doesn't exist there. Once the crash was fixed (a real file-existence guard per demo/channel, so a missing per-game HTML file skips that one demo directory instead of failing the whole deploy), stable's own "Play Breakout" link still 404'd, since decision 0046's whole design is that stable only ever moves via a deliberate, manual commit to STABLE_GLEMY_REF -- correctly working as designed, but the user explicitly rejected that design going forward: promotion should be automatic, not something a human has to remember to do, while still never shipping an untested or broken build as the public "stable" demo (the actual risk decision 0046 was protecting against).

Options considered

Collapse stable into edge entirely (STABLE_GLEMY_REF removed, stable always checks out glemy's current main) — rejected
Explicitly offered to and rejected by the user in favor of a CI-gated middle ground -- this option would remove decision 0046's actual protective value (a broken glemy push, even one that fails its own build/test, would immediately break the public stable demo too, not just edge) purely to satisfy 'automatic', when a real, still-safe automatic path exists.
Push-triggered cross-repo promotion: glemy's own CI, on success, sends a repository_dispatch event to glemy-website to promote immediately — rejected
Would need a personal access token with cross-repo write scope stored as a new secret in glemy -- the default GITHUB_TOKEN cannot trigger a workflow in a different repository. Not something to set up unilaterally on the user's behalf without them creating and handing over that credential. A scheduled poll in glemy-website achieves the same outcome (never promotes a commit whose CI hasn't passed) using only permissions glemy-website's own workflows already have -- reading a different *public* repository's Actions run history needs no authentication at all, confirmed directly against the real API before relying on it.
Let the auto-promotion commit's own `git push` (using GITHUB_TOKEN) trigger deploy.yml the normal way — rejected
Tried this first and it silently didn't work: the very first real auto-promotion run pushed STABLE_GLEMY_REF successfully, but no deploy.yml run ever fired for that commit. Root cause, confirmed via direct research: GitHub deliberately does not let a GITHUB_TOKEN-authored push trigger other workflows (an anti-recursion guard on push and other native events), with an explicit, documented exception for workflow_dispatch/repository_dispatch calls. Fixed by having promote-stable.yml explicitly call `gh workflow run deploy.yml` right after a real promotion, instead of relying on the push event -- required adding `actions: write` to the workflow's own permissions.

Decision

Two fixes, landed together as one real incident response: (1) .github/workflows/deploy.yml's per-game demo copy is now a guarded copy_demo() shell function -- checks the source HTML file actually exists in that checkout before creating the destination directory at all, so a game missing from a lagging stable checkout skips that one demo (falling back to build.gleam's own already-documented prepare_play_demo_dir behavior) instead of crashing the entire deploy. (2) Added glemy/.github/workflows/ci.yml (gleam build/gleam test on both targets, against a real headless WebGPU backend via lavapipe -- Mesa's software Vulkan implementation, the standard way to give Deno's native WebGPU, decision 0015, a real adapter to find with no GPU hardware present) as the first real CI glemy has ever had, giving 'this commit's build/test genuinely passed' a checkable GitHub status rather than only a local, manually-run gate (CLAUDE.md's own standing rule). Added glemy-website/.github/workflows/promote-stable.yml, superseding decision 0046's manual-only promotion: runs hourly (plus on-demand via workflow_dispatch), reads glemy's public Actions API for the most recent main commit with a passing CI run, and -- only if that differs from the current STABLE_GLEMY_REF -- commits the new SHA and explicitly dispatches deploy.yml (not a plain push, per the third rejected option above) to actually publish it. decisions.md/technical-architecture.md's §3.4 requirement ("promotion is a deliberate, explicit action, not an automatic consequence of a glemy push") is updated to reflect this: promotion is now automatic, but CI-gated, preserving the actual property decision 0046 valued (a real, auditable git commit records every promotion; an untested or failing build is never promoted) while removing the human-remembers-to-do-it failure mode that caused this whole incident.

Verification

Reproduced the original crash locally against a real scratch simulation (a source checkout with breakout.html present in 'edge' and genuinely absent in 'stable') before fixing it, confirmed the fix exits 0 and correctly skips only the affected demo directory, then confirmed a full local `gleam run -m build` still succeeds end-to-end with static/play-demo-breakout-stable genuinely missing (the Breakout stable link falls back to the documented unhashed path, as intended). glemy's new CI workflow was pushed and watched live: passed on the first real run (263 Erlang / 288 JavaScript tests, matching local counts exactly), confirming lavapipe genuinely gives headless Deno WebGPU a working adapter with no prior local reproduction needed. promote-stable.yml was pushed and manually dispatched twice: the first real run correctly found the latest green glemy commit and pushed STABLE_GLEMY_REF, but (as predicted by the GITHUB_TOKEN research above) no deploy.yml run followed -- confirmed live, not assumed, then fixed and re-verified with a second dispatch, after which deploy.yml genuinely ran and succeeded. The live site was checked directly afterward: /play's Breakout stable link now resolves to a real, content-hashed play-demo-breakout-stable-<hash>/ path (not the unhashed fallback), and both its index.html and game_breakout.mjs return real HTTP 200 responses from the actual deployed GitHub Pages site.

Consequences

glemy has real CI for the first time -- any future push to main that breaks gleam build/gleam test on either target is now visible as a failing GitHub check, not just something a local run might catch. Stable promotion is now genuinely hands-off: a future third game (RM-034) or any other change reaches the public stable demo automatically, within an hour of its own CI passing, with no separate 'remember to promote' step -- closing the exact failure mode this incident was caused by. deploy.yml's copy_demo() guard is reusable as-is for any future game's HTML entry point, so this specific crash class (a per-game file missing from a lagging checkout) cannot recur. The GITHUB_TOKEN/workflow_dispatch chaining requirement is now a documented, working pattern in this repo's own workflows, should a third workflow ever need to chain off either of these two.

References