01 - Quickstart

Turn one public site into agent-ready evidence.

Use maintained Node.js 22, 24, or 26. Start with finite budgets, inspect pages and failures together, then add browser, extraction, provider, or agent surfaces only where the job needs them.

terminal

npm install cockroach-crawler@0.7.0
npx cockroach-crawl https://example.com/docs \
  --max-pages 20 \
  --max-requests 80 \
  --max-duration 60000 \
  --jsonl \
  --output crawl.jsonl

02 - Workflow manuals

Start with the result you need.

03 - 50 capability pages

Navigate by category instead of scrolling through one wall.

Each capability now has a stable route with its purpose, prerequisite, public surface, quickstart, result contract, failure behavior, operating boundary, and previous or next link.

15 capabilitiesCrawl and discover

Choose seeds, traversal strategy, discovery rules, budgets, redirects, cancellation, caching, and compact maps.

Browse crawl ->09 capabilitiesRender and capture

Render JavaScript, perform bounded interaction, flatten readable component trees, and capture screenshot or PDF evidence.

Browse browser ->08 capabilitiesExtract agent-ready data

Produce readable Markdown, deterministic fields, PDF text, metadata, hashes, or schema-validated model output.

Browse extract ->06 capabilitiesReach public sources

Inspect provider capability, read public sources, and route optional official or operator-installed read-only providers.

Browse sources ->03 capabilitiesConnect agents and MCP

Expose strict crawler tools to agent runtimes through JavaScript, native MCP, or an optional Maqam gateway.

Browse agents ->04 capabilitiesDeploy and operate

Run the crawler through a token-authenticated Node or Docker API, a local playground, or a fixed-origin Worker profile.

Browse deploy ->05 capabilitiesKeep authority bounded

Admit public destinations, pin DNS decisions, validate redirects, enforce exact budgets, and stop at access challenges.

Browse security ->

Search 50 top-level capabilities Open typed reference

04 - Interfaces

Use the smallest interface that fits the host.

JavaScript

Typed library calls for applications, queues, test fixtures, and evidence pipelines.

Open JavaScript guide ->

CLI

Repeatable local exports, CI jobs, scheduled snapshots, and content inventories.

Open CLI guide ->

MCP

Native read-only tools for Codex, Claude Code, and other stdio MCP clients.

Open MCP guide ->

Docker or Worker

Token-authenticated service deployment or a smaller fixed-origin edge fetch profile.

Open deployment guide ->

The complete web toolkit for JavaScript AI agents.

Turn one public site into agent-ready evidence.

Start with the result you need.

Navigate by category instead of scrolling through one wall.

Use the smallest interface that fits the host.