Skip to content

fix(railway): watchdog never paginates — GitHub rewrites next links to the repository-ID path (#6378) - #6469

Open
koala73 wants to merge 4 commits into
mainfrom
fix/railway-watchdog-pagination-6378
Open

fix(railway): watchdog never paginates — GitHub rewrites next links to the repository-ID path (#6378)#6469
koala73 wants to merge 4 commits into
mainfrom
fix/railway-watchdog-pagination-6378

Conversation

@koala73

@koala73 koala73 commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Found while executing #6378. Provisioning the watchdog HMAC removed the failure that was masking this one.

The bug

#paginate in scripts/dispatch-stale-railway-reconcile.mjs validated GitHub's rel="next" URL against a synthesized /repos/<owner>/<repo>/... path and threw when they differed. GitHub always rewrites paginated Link headers to the numeric repository-ID form. Reproduced against the live API:

requested: /repos/koala73/worldmonitor/commits/<sha>/statuses?per_page=100&page=1
next link: /repositories/1130564872/statuses/<sha>?per_page=100&page=2

actualNext.pathname !== expectedNext.pathnameGITHUB_READ_FAILED: GitHub collection next link was not exact.

All three paginated collections are affected (commit statuses, workflow runs, attempt jobs). Main's head carries more than 100 statuses, so readNewestGate always saw a next link and always threw — the watchdog returned DEFERRED_AMBIGUOUS on every scheduled run and could never have classified or dispatched a recovery after the cutover.

Observed in production run 31496029981:

##[warning]DEFERRED_AMBIGUOUS: GitHub collection next link was not exact

Why tests missed it

Every mock built its next link from the request pathname (json(..., { link: https://api.github.com${parsed.pathname}?...page=2 })), echoing back the one form the real API never returns. The assertion could not fail.

The fix

Treat the origin's link as nothing but a boolean "another page exists" and synthesize the next request from our own allowlisted path. The origin no longer routes a watchdog request at all — which also keeps #authorize meaningful, since the old code fed it an absolute /repositories/<id>/... URL it would have rejected as GITHUB_ENDPOINT_FORBIDDEN had the "not exact" check not fired first.

Completeness guarantees are unchanged: frozen total_count, duplicate-id detection, truncation-without-next, next-after-total, and the WATCHDOG_MAX_PAGES budget all still apply.

Tests

  • New test models the real repository-ID rewrite and asserts page 2 is fetched on the canonical path. Proven red before the fix with the exact production error, green after.
  • The former 'skipped next page' rejection case is unreachable once the walk is synthesized, so it is replaced by a positive test proving an origin-advertised page number is ignored while every page is still walked.
  • node --test tests/railway-stale-reconcile-dispatch.test.mjs → 51/51 pass.
  • Workflow contract tests → 21/21 pass. Biome clean.

Also

Corrects the runbook provisioning table, which omitted the verifier HMAC from ingestion-acceptance-production-verification and had no row for ingestion-acceptance-production.

Refs #6378

https://claude.ai/code/session_01GYUZ5eKm69sim6cEHpGJRh

…o the repository-ID path

`#paginate` compared GitHub's `rel="next"` URL against a synthesized
`/repos/<owner>/<repo>/...` path and threw when they differed. GitHub always
rewrites paginated Link headers to the numeric repository-ID form
(`/repositories/1130564872/statuses/<sha>`), so the comparison never matched
and every multi-page read failed closed with `next link was not exact`.

Main's head carries more than 100 statuses, so `readNewestGate` always saw a
next link and always threw. The watchdog therefore returned DEFERRED_AMBIGUOUS
on every scheduled run while reporting success, and could never have
classified or dispatched a recovery after the cutover.

Every mock in the suite built its next link from the request pathname, so the
assertion echoed back the one form the real API never returns and the tests
could not fail.

Follow the origin's link only as a boolean "another page exists" and synthesize
the next request from our own allowlisted path. The origin no longer routes a
watchdog request at all, which also keeps the `#authorize` allowlist meaningful
— the old code fed it an absolute `/repositories/<id>/...` URL it would have
rejected as GITHUB_ENDPOINT_FORBIDDEN. Completeness stays enforced by the frozen
total_count, duplicate-id, and truncation invariants.

The former 'skipped next page' rejection case is unreachable now that the walk
is synthesized; it is replaced by a positive test proving an origin-advertised
page number is ignored and every page is still walked.

Refs #6378
… reads

The provisioning table omitted the verifier HMAC from
`ingestion-acceptance-production-verification`, even though that environment's
`verifier` job in railway-deploy-trigger.yml reads
`secrets.RAILWAY_RECONCILE_VERIFIER_HMAC`, and it listed no row at all for
`ingestion-acceptance-production` and its mutation HMAC.

Refs #6378
@mintlify

mintlify Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
WorldMonitor 🟢 Ready View Preview Aug 11, 2026, 1:43 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@vercel

vercel Bot commented Aug 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Aug 11, 2026 6:19pm

Request Review

Railway cannot issue a read-only token. `apiTokenCreate` takes only
`{name, workspaceId}` and inherits the creating user's permissions;
`projectTokenCreate` takes only `{projectId, environmentId, name}`. The VIEWER
value on ProjectRole/TeamRole applies to project members — user accounts — not
to tokens, so a genuinely read-only token needs a separate Railway account
joined as a VIEWER member.

The runbook promised "The Viewer token must not be able to deploy, redeploy,
edit configuration, or approve Railway work." That is a credential-level
guarantee no Railway token provides, and the preview, verifier, and proof jobs
are all built on it. Say instead where the property actually comes from: the
--workflow-authorized fence in scripts/trigger-railway-deploys.mjs and the
workflow contract tests.

Also records that projectTokenCreate rejects a CLI session with Not Authorized,
so both tokens must be minted from the dashboard.

Refs #6378
…erson

The provisioning table asked for "independent reviewers and prevent self-review"
on ingestion-acceptance-production-breakglass. The environment now requires a
reviewer, so resolve pauses for a human and GitHub records the approver, but
prevent_self_review is deliberately off.

Enabling it with one reviewer would deadlock the emergency path: the only person
able to dispatch recovery would be barred from approving it, precisely when
recovery is needed. Two-person control requires adding a second named reviewer
first, so record that the `approver` input remains an audit assertion rather
than a verified second party.

Refs #6378
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant