POST /v1/web/crawl starts from a URL, finds the site’s pages and reads each one. It runs as a background job: you get a crawlId straight away, then poll it or let a webhook tell you as pages finish. 1 credit per page read.
When to use it
- You want every page of a site, or of one section, and you do not have the list of URLs.
- The job is too big to wait for in one request.
Example
Keeping a crawl inside one section
A crawl stays inside the section of the site its starting URL is in. A crawl started athttps://example.com/blog reads /blog and the pages under it, even though the site’s published page list (its sitemap) covers the whole site. A crawl started at the homepage covers the whole site.
To read the whole site from a deeper starting page, set crawlEntireDomain:
includePaths replaces the section with your own patterns, written so the starting page matches them too, such as ["^/blog", "^/news"]. If your patterns would exclude the starting page, the crawl has nowhere to begin, so the request is refused instead of starting a job that finds nothing.
Options
Only follow URLs whose path matches every one of these patterns, such as
["^/blog"]. Up to 50. They apply to the starting URL too.Skip URLs whose path matches any of these patterns, such as
["^/blog/tag"]. Up to 50. They also apply to the starting URL.Read the whole site instead of only the section the starting URL is in. See Keeping a crawl inside one section.
Whether to use the site’s published page list to find pages:
include or skip. Pages it lists outside the crawl’s section are left out.How to read each page. The same options as a Scrape, with formats limited to
markdown, html, rawHtml, links and structured.Have us call your URL as pages finish and when the job ends, instead of polling. See Jobs & webhooks.
Getting the results
PollGET /v1/web/crawl/{id} for progress and each finished page. Results stay available for 90 days after the crawl last changes. After that the call answers 404.
Each page with its
url, status (queued, running, completed or failed), depth, discoveredVia and data once it is read. A failed page also has error, an errorCode and retryable, which says whether asking again could help. requestedUrl is the URL as you gave it, and appears only when it differs from url after we normalised it.discoveredVia says how the URL got into the job: requested for the starting URL, sitemap for one from the site’s page list, and link for one followed from a page we read. It is the quickest way to see why a crawl returned what it did, and whether a higher limit or maxDepth would find more.Cancelling a crawl
DELETE /v1/web/crawl/{id} stops a running crawl. Pages already read keep their results and their charge. Pages not yet read are marked cancelled and cost nothing. Cancelling a crawl that already finished returns its status unchanged.
Cost
1 credit per page read. Pages that fail are free. Setlimit, maxDepth and path filters to keep the cost predictable, and check the reach first with Map.
Next steps
Map a site
See a crawl’s reach before reading anything.
Read a list of URLs
When you already have the URLs.
Jobs & webhooks
Polling, webhooks and checking deliveries.
API reference
The full schema, with a live playground.