Skip to content

fix(build): Optimize incremental builds with shared content graph hashing - #17630

Open
ericclemmons wants to merge 4 commits into
withastro:mainfrom
ericclemmons:codex/incremental-graph-hashing
Open

fix(build): Optimize incremental builds with shared content graph hashing#17630
ericclemmons wants to merge 4 commits into
withastro:mainfrom
ericclemmons:codex/incremental-graph-hashing

Conversation

@ericclemmons

@ericclemmons ericclemmons commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Changes

Optimizes experimental cache from O(R × G) to O(G × R) using a Merkle-style hashing algorithm to content roots that heavily share dependencies.

  • R = # of roots. (e.g. 8,000 content pages)
  • G = graph size.
flowchart LR
    subgraph Before["Before: O(R × G)"]
        P1["Entry 1"] --> G1["Walk graph"]
        P2["Entry 2"] --> G2["Walk graph"]
        P3["Entry 3"] --> G3["Walk graph"]
    end

    subgraph After["After: O(G + R)"]
        G["Analyze graph once"] --> H["Hash cache"]
        H --> R1["Entry 1: O(1) lookup"]
        H --> R2["Entry 2: O(1) lookup"]
        H --> R3["Entry 3: O(1) lookup"]
    end
Loading

Testing

The performance degradation was discovered on my fork of cloudflare-docs where I enabled incrementalBuild. ~10m builds now timed out at 30 minutes.

  • TODO: Can this be published on pkg.pr.new to validate?
13:01:47.975
> astro build
13:01:47.976
13:01:58.504
18:01:58 [astro-skills] Setting up Agent Skills Discovery routes
13:01:58.504
18:01:58 [astro-skills] Agent Skills Discovery routes configured
13:02:01.502
18:02:01 [build] Waiting for integration "@cloudflare/nimbus-docs", hook "astro:config:setup"...
13:02:14.225
18:02:14 [WARN] [@cloudflare/nimbus-docs] [nimbus-docs] (nimbus/duplicate-slug, warning) 3 routes are served by an explicit src/pages file that shadows a content entry at the same URL:
13:02:14.225
  /ai/models  ←  src/content/docs/ai/models/index.mdx, src/pages/ai/models/index.astro
13:02:14.225
  /ruleset-engine/rules-language/fields/reference  ←  src/content/docs/ruleset-engine/rules-language/fields/reference/index.mdx, src/pages/ruleset-engine/rules-language/fields/reference/index.astro
13:02:14.225
  /workers-ai/models  ←  src/content/docs/workers-ai/models/index.mdx, src/pages/workers-ai/models/index.astro
13:02:14.225
13:02:14.226
Astro serves the page and drops the content route (deterministic). Intended when a content page wraps a custom page component; verify each shadow is intentional.
13:02:18.146
18:02:18 [content] Syncing content
13:02:18.149
18:02:18 [content] Astro config changed
13:02:18.150
18:02:18 [content] Clearing content store
13:02:28.869
18:02:28 [skills-loader] Loaded 11 skill(s) from "skills"
13:02:41.475
18:02:41 [content] Synced content
13:02:41.488
18:02:41 [types] Generated 27.00s
13:02:41.489
18:02:41 [build] output: "static"
13:02:41.490
18:02:41 [build] mode: "static"
13:02:41.490
18:02:41 [build] directory: /opt/buildhome/repo/dist/
13:02:41.490
18:02:41 [build] Collecting build info...
13:02:41.495
18:02:41 [build] ✓ Completed in 43.00s.
13:02:41.498
18:02:41 [build] Building static entrypoints...
13:04:25.008
18:04:25 [astro-icon] Loaded icons from src/icons, ph, simple-icons, vscode-icons
13:31:14.749
Build took too long and was timed out

Docs

@changeset-bot

changeset-bot Bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a4ac9dc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 417 packages
Name Type
astro Patch
@e2e/astro-linked-lib Patch
@e2e/actions-blog Patch
@e2e/actions-react-19 Patch
@e2e/astro-component Patch
@e2e/astro-envs Patch
@e2e/astro-island-hydration-error Patch
@test/astro-cloudflare-node-prerender-mdx Patch
@test/astro-cloudflare Patch
@e2e/content-collections Patch
@e2e/csp-server-islands Patch
@e2e/css Patch
@test/custom-client-directives Patch
@e2e/dev-toolbar Patch
@e2e/error-cyclic Patch
@e2e/error-sass Patch
@e2e/errors Patch
@e2e/hydration-race Patch
@e2e/i18n Patch
@test/nested-style-bug-e22e Patch
@e2e/preact-compat-component Patch
@e2e/preact-component Patch
@e2e/preact-lazy-component Patch
@e2e/prefetch Patch
@e2e/react-component Patch
@e2e/server-islands-key Patch
@e2e/server-islands Patch
@e2e/solid-circular Patch
@e2e/solid-component Patch
@e2e/solid-recurse Patch
@e2e/svelte-component Patch
@e2e/e2e-tailwindcss Patch
@e2e/ts-resolution Patch
@e2e/view-transitions Patch
@e2e/vue-component Patch
@performance/md Patch
@performance/mdoc Patch
@performance/mdx Patch
@test/0-css Patch
fake-astro-library Patch
@test/actions Patch
@test/alias-css-url-data-uri Patch
@test/alias-css-url Patch
@test/alias-path-alias-style Patch
@test/ts-paths-no-baseurl Patch
@test/aliases-tsconfig Patch
@test/aliases Patch
@test/api-routes Patch
@test/asset-query-params-chunks Patch
@test/asset-url-base Patch
@test/astro-pages Patch
@test/astro-assets-prefix Patch
@test/astro-assets Patch
@test/astro-basic Patch
@test/astro-check-errors Patch
@test/astro-check-no-errors Patch
@test/astro-check-watch Patch
@test/astro-children Patch
@test/astro-client-only Patch
@test/astro-component-bundling Patch
@test/astro-component-code Patch
@test/astro-css-bundling Patch
@test/astro-dev-headers Patch
@test/astro-dev-http2 Patch
@test/astro-doctype Patch
@test/astro-dynamic Patch
@test/astro-env-content-collections Patch
@test/astro-env-required-public Patch
@test/astro-env-server-fail Patch
@test/astro-env-server-secret Patch
@test/astro-env Patch
@test/astro-envs Patch
@test/astro-expr Patch
@test/astro-get-static-paths Patch
@test/astro-head Patch
@test/astro-manifest-client-script Patch
@test/astro-manifest-invalid Patch
@test/astro-manifest Patch
@test/astro-markdown-frontmatter-injection Patch
@test/astro-markdown-plugins Patch
@test/astro-markdown-remarkRehype Patch
@test/astro-markdown-skiki-default-color Patch
@test/astro-markdown-skiki-langs Patch
@test/astro-markdown-skiki-themes-custom Patch
@test/astro-markdown-skiki-themes-integrated Patch
@test/astro-markdown-skiki-wrap-false Patch
@test/astro-markdown-skiki-wrap-null Patch
@test/astro-markdown-skiki-wrap-true Patch
@test/astro-markdown-url Patch
@test/astro-markdown Patch
@test/astro-mode Patch
@test/astro-page-directory-url Patch
@test/astro-partial-html Patch
@test/astro-preview-allowed-hosts Patch
@test/astro-preview-headers Patch
@test/astro-public Patch
@test/astro-script-template-dedup Patch
@test/astro-scripts Patch
@test/astro-slots-nested Patch
@test/concurrency Patch
@test/build-readonly-file Patch
@test/cache-memory-query-include Patch
@test/cache-memory-query Patch
@test/client-address-node Patch
@test/client-only-child-styles Patch
@test/client-only-css-chunk-leak Patch
@test/code-component Patch
@test/component-library Patch
@test/config-vite-css-target-no-minify Patch
@test/config-vite-css-target Patch
@test/config-vite Patch
@test/react-container Patch
@test/content-with-spaces-in-folder-name Patch
@test/content-collection-picture-render Patch
@example/content-collection-references Patch
@test/content-collection-tla-svg Patch
@test/content-collections-base Patch
@test/content-collections-empty-dir Patch
@test/content-collections-empty-md-file Patch
@test/content-collections-image-hmr Patch
@test/content-collections-mutation Patch
@test/content-collections-number-id Patch
@test/content-collections-type-inference Patch
@test/content-collections-with-config-mjs Patch
@test/content-collections Patch
@test/content-frontmatter Patch
@test/content-intellisense Patch
@test/content-layer-loader-schema-function Patch
@test/content-layer-remark-plugins Patch
@test/content-layer Patch
@test/content-ssr-integration Patch
@test/content-static-paths-integration Patch
@test/content Patch
@test/core-image-data-url Patch
@test/core-image-deletion-ssr Patch
@test/core-image-deletion Patch
@test/core-image-errors Patch
@test/core-image-fs-config Patch
@test/core-image-remark-infersize Patch
@test/core-image-layout Patch
@test/core-image-picture-emit-file Patch
@test/core-image-remark-imgattr Patch
@test/core-image-ssg Patch
@test/core-image-ssr Patch
@test/core-image-svg-in-client Patch
@test/core-image-svg Patch
@test/core-image-unconventional-settings Patch
@test/core-image Patch
@test/csp-adapter Patch
@test/csp-fonts Patch
@test/csp Patch
@test/css-assets Patch
@test/css-dangling-references Patch
@test/css-deduplication Patch
@test/css-double-bundle Patch
@test/css-dynamic-import-dev Patch
@test/css-import-as-inline Patch
@test/css-inline-stylesheets Patch
@test/css-no-code-split Patch
@test/css-path-case Patch
@test/css-pure-chunk-query-params Patch
@test/css-server-output-dedup Patch
@test/custom-404-injected-from-dep Patch
@test/custom-404-pkg Patch
custom-fetch-error-pages Patch
@test/custom-renderer Patch
@test/data-collections-schema Patch
@test/data-collections Patch
@test/debug-component Patch
@test/dev-container Patch
@test/dev-render Patch
@test/dev-request-url Patch
@test/dynamic-endpoint-collision Patch
@test/dynamic-route-build-file Patch
@test/endpoint-routing Patch
@test/error-bad-js Patch
@test/error-build-location Patch
@test/error-non-error Patch
@test/extension-matching Patch
@test/fetch Patch
@test/fonts Patch
@test/astro-fontsource-package Patch
@test/get-static-paths-pages Patch
@test/glob-pages-css Patch
@test/head-propagation-prerender-env Patch
@test/hmr-markdown Patch
@test/hmr-middleware Patch
@test/hmr-new-page Patch
@test/hmr-slots-render Patch
@test/hoisted-imports Patch
@test/html-component Patch
@test/html-escape Patch
@test/html-page Patch
@test/html-slots Patch
@test/hydration-race Patch
@test/i18n-client-import Patch
@test/i18n-css-leak-basic Patch
@test/import-ts-with-js Patch
@test/impostor-md-file Patch
@test/incremental-build-client-only Patch
@test/incremental-build-concurrency Patch
@test/incremental-build-content Patch
@test/incremental-build-drop-key Patch
@test/incremental-build-dynamic-import Patch
@test/incremental-build-headers Patch
@test/incremental-build-images Patch
@test/incremental-build-islands Patch
@test/incremental-build-no-output Patch
@test/incremental-build-script Patch
@test/incremental-build Patch
@test/integration-add-page-extension Patch
@test/integration-server-setup Patch
@test/jsx-queue-rendering Patch
@test/large-array-solid Patch
@test/legacy-collections-backwards-compat Patch
@test/lightningcss-css-modules-content Patch
@test/lightningcss-scoped-nesting Patch
@test/live-loaders Patch
@test/markdown Patch
@test/middleware-dev Patch
@test/middleware-full-ssr Patch
@test/middleware-no-user-middlewaqre Patch
@test/middleware-tailwind Patch
@test/minification-html-default Patch
@test/minification-html-jsx Patch
@test/minification-html Patch
@test/non-ascii-path Patch
@test/non-html-pages Patch
@test/page-format Patch
@test/page-level-styles Patch
@test/parallel-components Patch
@test/partials-css-boundary Patch
@test/partials Patch
@test/passthrough-image-service Patch
@test/postcss Patch
@test/preact-compat-component Patch
@test/preact-component Patch
@test/remote-css Patch
@test/request-signal Patch
@test/reuse-injected-entrypoint Patch
@test/root-srcdir-css Patch
@test/scoped-style-strategy Patch
@test/server-entry-fake-adapter Patch
@test/server-entry Patch
@test/server-islands-hybrid Patch
@test/server-islands-ssr Patch
@test/session-false Patch
@test/session-tree-shake Patch
@test/sessions Patch
@test/slots-preact Patch
@test/slots-react Patch
@test/slots-solid Patch
@test/slots-svelte Patch
@test/slots-vue Patch
@test/solid-component Patch
@test/sourcemap Patch
@test/space-in-folder-name Patch
@test/special-chars-in-component-imports Patch
@test/ssr-assets Patch
@test/ssr-dynamic Patch
@test/ssr-partytown Patch
@test/ssr-prerender-get-static-paths Patch
@test/ssr-prerender Patch
@test/ssr-preview Patch
@test/ssr-renderers-static-vue Patch
@test/ssr-request Patch
@test/ssr-hoisted-script Patch
@test/ssr-scripts Patch
@test/static-build-code-component Patch
@test/static-build-dir Patch
@test/static-build-frameworks Patch
@test/static-build-page-url-format Patch
@test/static-build-ssr Patch
@test/static-build Patch
@test/static-redirect Patch
@test/svelte-component Patch
@test/svg-deduplication Patch
@test/tailwindcss Patch
@e2e/third-party-astro Patch
@test/url-import-suffix Patch
@test/view-transitions Patch
@test/virtual-astro-file Patch
@test/vite-virtual-modules Patch
@test/vitest Patch
@test/vue-component Patch
@test/vue-with-multi-renderer Patch
@test/alpinejs-basics Patch
@test/alpinejs-directive Patch
@test/alpinejs-plugin-script-import Patch
@test/astro-cloudflare-allowed-hosts Patch
@test/astro-cloudflare-astro-dev-platform Patch
@test/astro-cloudflare-astro-env Patch
@test/astro-cloudflare-binding-image-cache Patch
@test/astro-cloudflare-binding-image-service Patch
@test/astro-cloudflare-cache-provider-wait-until Patch
@test/astro-cloudflare-cache-provider Patch
@test/astro-cloudflare-client-address Patch
@test/astro-cloudflare-compile-custom-image-service Patch
@test/astro-cloudflare-compile-image-service Patch
@test/astro-cloudflare-custom-entryfile Patch
@test/astro-cloudflare-custom-image-service Patch
@test/astro-cloudflare-dev-image-endpoint Patch
@test/astro-cloudflare-external-image-service Patch
@test/astro-cloudflare-external-redirects Patch
@test/cloudflare-incremental-images Patch
@test/astro-cloudflare-internal-redirects Patch
@test/astro-cloudflare-no-output Patch
@test/astro-cloudflare-prerender-node-env Patch
@test/astro-cloudflare-prerender-queue-consumers Patch
@test/astro-cloudflare-prerender-styles Patch
@test/astro-cloudflare-prerenderer-errors Patch
@test/astro-cloudflare-prerenderer-render-error Patch
@test/routing-priority-cloudflare Patch
@test/cf-server-entry Patch
@test/astro-cloudflare-server-island-prerender-framework Patch
@test/astro-cloudflare-sql-import Patch
@test/cf-ssr-deps Patch
@test/astro-cloudflare-static Patch
@test/astro-cloudflare-svelte-rune-deps Patch
@test/astro-cloudflare-top-level-return Patch
@test/cf-user-optimize-deps Patch
@test/astro-cloudflare-vite-plugin Patch
@test/astro-cloudflare-with-base Patch
@test/astro-cloudflare-with-react Patch
@test/astro-cloudflare-with-solid-js Patch
@test/astro-cloudflare-with-svelte Patch
@test/astro-cloudflare-with-vue Patch
@test/astro-cloudflare-wrangler-preview-platform Patch
@test/markdoc-content-collections Patch
@test/content-layer-markdoc Patch
@test/headings-custom Patch
@test/headings Patch
@test/image-assets-custom Patch
@test/image-assets Patch
@test/markdoc-propagated-assets Patch
@test/markdoc-render-with-space Patch
@test/markdoc-render-html Patch
@test/markdoc-render-null Patch
@test/markdoc-render-partials Patch
@test/markdoc-render-simple Patch
@test/markdoc-render-table-attrs Patch
@test/markdoc-render-this-context Patch
@test/markdoc-render-typographer Patch
@test/markdoc-render-with-components Patch
@test/markdoc-render-with-config Patch
@test/markdoc-render-with-extends-components Patch
@test/markdoc-render-with-indented-components Patch
@test/markdoc-render-with-transform Patch
@test/markdoc-variables Patch
@test/content-layer-rendering Patch
@test/mdx-css-head-mdx Patch
@test/image-remark-imgattr Patch
@test/mdx-astro-container-escape Patch
@test/mdx-frontmatter-injection Patch
@test/netlify-skew-protection Patch
@test/netlify-hosted-astro-project Patch
@test/nodejs-api-route Patch
@test/nodejs-badurls Patch
@test/nodejs-encoded Patch
@test/nodejs-errors Patch
@test/nodejs-headers Patch
@test/nodejs-image Patch
@test/locals Patch
@test/node-middleware Patch
@test/nodejs-prerender-404-500 Patch
@test/nodejs-prerender Patch
@test/nodejs-prerendered-error-page-fetch Patch
@test/nodejs-preview-headers Patch
@test/redirects Patch
@test/node-sessions Patch
@test/ssr-assets-middleware Patch
@test/node-static-headers Patch
@test/node-trailingslash Patch
@test/url Patch
@test/well-known-locations Patch
@test/react-component Patch
@test/sitemap-chunks Patch
@test/sitemap-dynamic Patch
@test/sitemap-i18n-fallback Patch
@test/sitemap-ssr Patch
@test/sitemap-static Patch
@test/sitemap-trailing-slash Patch
async-rendering Patch
conditional-rendering Patch
@test/empty-class Patch
svelte-prop-types Patch
@test/astro-vercel-basic Patch
@test/astro-vercel-image Patch
@test/astro-vercel-integration-assets Patch
@test/vercel-isr Patch
@test/vercel-max-duration Patch
@test/vercel-edge-middleware-with-edge-file Patch
@test/vercel-edge-middleware-without-edge-file Patch
@test/astro-vercel-no-output Patch
@test/astro-vercel-prerendered-error-pages Patch
@test/astro-vercel-redirects-serverless Patch
@test/astro-vercel-redirects Patch
@test/vercel-server-islands Patch
@test/astro-vercel-serverless-prerender Patch
@test/astro-vercel-serverless-with-dynamic-routes Patch
@test/astro-vercel-static-assets Patch
@test/vercel-static-headers Patch
@test/astro-vercel-static Patch
@test/vercel-streaming Patch
@test/astro-vercel-with-web-analytics-enabled-output-as-static Patch
vercel-hosted-astro-project Patch
@test/vue-app-entrypoint-async Patch
@test/vue-app-entrypoint-css Patch
@test/vue-app-entrypoint-no-export-default Patch
@test/vue-app-entrypoint-relative Patch
@test/vue-app-entrypoint-src-absolute Patch
@test/vue-app-entrypoint Patch
@test/vue-basics Patch
vue-prop-types Patch
astro-benchmark Patch
@benchmark/adapter Patch
@benchmark/timer Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the pkg: astro Related to the core `astro` package (scope) label Aug 7, 2026
@codspeed-hq

codspeed-hq Bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 21 untouched benchmarks

Comparing ericclemmons:codex/incremental-graph-hashing (a4ac9dc) with main (630b833)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (d081033) during the generation of this report, so 630b833 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@ericclemmons ericclemmons changed the title fix(build): avoid repeated content graph hashing fix(build): Optimize incremental builds with shared content graph hashing Aug 7, 2026
@matthewp matthewp added the pr preview Apply this label to a PR to generate a preview release label Aug 7, 2026
@ericclemmons

ericclemmons commented Aug 7, 2026

Copy link
Copy Markdown
Author

Closed in favor of #17632 so I can do a pr preview

@ericclemmons ericclemmons reopened this Aug 11, 2026
@ericclemmons ericclemmons mentioned this pull request Aug 11, 2026
Closed
@ericclemmons
ericclemmons marked this pull request as ready for review August 11, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

pkg: astro Related to the core `astro` package (scope) pr preview Apply this label to a PR to generate a preview release

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ericclemmons @matthewp