fix(railway): watchdog never paginates — GitHub rewrites next links to the repository-ID path (#6378) - #6469
Open
koala73 wants to merge 4 commits into
Open
fix(railway): watchdog never paginates — GitHub rewrites next links to the repository-ID path (#6378)#6469koala73 wants to merge 4 commits into
koala73 wants to merge 4 commits into
Conversation
…o the repository-ID path `#paginate` compared GitHub's `rel="next"` URL against a synthesized `/repos/<owner>/<repo>/...` path and threw when they differed. GitHub always rewrites paginated Link headers to the numeric repository-ID form (`/repositories/1130564872/statuses/<sha>`), so the comparison never matched and every multi-page read failed closed with `next link was not exact`. Main's head carries more than 100 statuses, so `readNewestGate` always saw a next link and always threw. The watchdog therefore returned DEFERRED_AMBIGUOUS on every scheduled run while reporting success, and could never have classified or dispatched a recovery after the cutover. Every mock in the suite built its next link from the request pathname, so the assertion echoed back the one form the real API never returns and the tests could not fail. Follow the origin's link only as a boolean "another page exists" and synthesize the next request from our own allowlisted path. The origin no longer routes a watchdog request at all, which also keeps the `#authorize` allowlist meaningful — the old code fed it an absolute `/repositories/<id>/...` URL it would have rejected as GITHUB_ENDPOINT_FORBIDDEN. Completeness stays enforced by the frozen total_count, duplicate-id, and truncation invariants. The former 'skipped next page' rejection case is unreachable now that the walk is synthesized; it is replaced by a positive test proving an origin-advertised page number is ignored and every page is still walked. Refs #6378
… reads The provisioning table omitted the verifier HMAC from `ingestion-acceptance-production-verification`, even though that environment's `verifier` job in railway-deploy-trigger.yml reads `secrets.RAILWAY_RECONCILE_VERIFIER_HMAC`, and it listed no row at all for `ingestion-acceptance-production` and its mutation HMAC. Refs #6378
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Railway cannot issue a read-only token. `apiTokenCreate` takes only
`{name, workspaceId}` and inherits the creating user's permissions;
`projectTokenCreate` takes only `{projectId, environmentId, name}`. The VIEWER
value on ProjectRole/TeamRole applies to project members — user accounts — not
to tokens, so a genuinely read-only token needs a separate Railway account
joined as a VIEWER member.
The runbook promised "The Viewer token must not be able to deploy, redeploy,
edit configuration, or approve Railway work." That is a credential-level
guarantee no Railway token provides, and the preview, verifier, and proof jobs
are all built on it. Say instead where the property actually comes from: the
--workflow-authorized fence in scripts/trigger-railway-deploys.mjs and the
workflow contract tests.
Also records that projectTokenCreate rejects a CLI session with Not Authorized,
so both tokens must be minted from the dashboard.
Refs #6378
…erson The provisioning table asked for "independent reviewers and prevent self-review" on ingestion-acceptance-production-breakglass. The environment now requires a reviewer, so resolve pauses for a human and GitHub records the approver, but prevent_self_review is deliberately off. Enabling it with one reviewer would deadlock the emergency path: the only person able to dispatch recovery would be barred from approving it, precisely when recovery is needed. Two-person control requires adding a second named reviewer first, so record that the `approver` input remains an audit assertion rather than a verified second party. Refs #6378
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while executing #6378. Provisioning the watchdog HMAC removed the failure that was masking this one.
The bug
#paginateinscripts/dispatch-stale-railway-reconcile.mjsvalidated GitHub'srel="next"URL against a synthesized/repos/<owner>/<repo>/...path and threw when they differed. GitHub always rewrites paginated Link headers to the numeric repository-ID form. Reproduced against the live API:actualNext.pathname !== expectedNext.pathname→GITHUB_READ_FAILED: GitHub collection next link was not exact.All three paginated collections are affected (commit statuses, workflow runs, attempt jobs). Main's head carries more than 100 statuses, so
readNewestGatealways saw a next link and always threw — the watchdog returnedDEFERRED_AMBIGUOUSon every scheduled run and could never have classified or dispatched a recovery after the cutover.Observed in production run 31496029981:
Why tests missed it
Every mock built its next link from the request pathname (
json(..., { link:https://api.github.com${parsed.pathname}?...page=2})), echoing back the one form the real API never returns. The assertion could not fail.The fix
Treat the origin's link as nothing but a boolean "another page exists" and synthesize the next request from our own allowlisted path. The origin no longer routes a watchdog request at all — which also keeps
#authorizemeaningful, since the old code fed it an absolute/repositories/<id>/...URL it would have rejected asGITHUB_ENDPOINT_FORBIDDENhad the "not exact" check not fired first.Completeness guarantees are unchanged: frozen
total_count, duplicate-id detection, truncation-without-next, next-after-total, and theWATCHDOG_MAX_PAGESbudget all still apply.Tests
'skipped next page'rejection case is unreachable once the walk is synthesized, so it is replaced by a positive test proving an origin-advertised page number is ignored while every page is still walked.node --test tests/railway-stale-reconcile-dispatch.test.mjs→ 51/51 pass.Also
Corrects the runbook provisioning table, which omitted the verifier HMAC from
ingestion-acceptance-production-verificationand had no row foringestion-acceptance-production.Refs #6378
https://claude.ai/code/session_01GYUZ5eKm69sim6cEHpGJRh