Skip to main content
POST /v1/web/crawl starts from a URL, finds the site’s pages and reads each one. It runs as a background job: you get a crawlId straight away, then poll it or let a webhook tell you as pages finish. 1 credit per page read.

When to use it

  • You want every page of a site, or of one section, and you do not have the list of URLs.
  • The job is too big to wait for in one request.
If you already have the URLs, use Batch. To see which URLs a crawl would reach without reading them, use Map.

Example

The crawl starts at once:

Keeping a crawl inside one section

A crawl stays inside the section of the site its starting URL is in. A crawl started at https://example.com/blog reads /blog and the pages under it, even though the site’s published page list (its sitemap) covers the whole site. A crawl started at the homepage covers the whole site. To read the whole site from a deeper starting page, set crawlEntireDomain:
includePaths replaces the section with your own patterns, written so the starting page matches them too, such as ["^/blog", "^/news"]. If your patterns would exclude the starting page, the crawl has nowhere to begin, so the request is refused instead of starting a job that finds nothing.

Options

string
required
Where to start. Must be http or https.
number
default:"100"
The most pages to read, from 1 to 500.
number
default:"3"
How many links away from the starting page to follow, from 0 to 10.
string[]
Only follow URLs whose path matches every one of these patterns, such as ["^/blog"]. Up to 50. They apply to the starting URL too.
string[]
Skip URLs whose path matches any of these patterns, such as ["^/blog/tag"]. Up to 50. They also apply to the starting URL.
boolean
default:"false"
Read the whole site instead of only the section the starting URL is in. See Keeping a crawl inside one section.
boolean
default:"false"
Also follow links into the site’s subdomains.
Also follow links that leave the site.
boolean
default:"false"
Treat URLs that differ only in their query string as the same page.
string
default:"include"
Whether to use the site’s published page list to find pages: include or skip. Pages it lists outside the crawl’s section are left out.
object
How to read each page. The same options as a Scrape, with formats limited to markdown, html, rawHtml, links and structured.
object
Have us call your URL as pages finish and when the job ends, instead of polling. See Jobs & webhooks.

Getting the results

Poll GET /v1/web/crawl/{id} for progress and each finished page. Results stay available for 90 days after the crawl last changes. After that the call answers 404.
string
The job’s state: running, completed, failed or cancelled.
number
How many pages the job holds.
number
Pages read so far.
number
Pages that could not be read.
number
Credits used so far, 1 per page read.
object[]
Each page with its url, status (queued, running, completed or failed), depth, discoveredVia and data once it is read. A failed page also has error, an errorCode and retryable, which says whether asking again could help. requestedUrl is the URL as you gave it, and appears only when it differs from url after we normalised it.discoveredVia says how the URL got into the job: requested for the starting URL, sitemap for one from the site’s page list, and link for one followed from a page we read. It is the quickest way to see why a crawl returned what it did, and whether a higher limit or maxDepth would find more.

Cancelling a crawl

DELETE /v1/web/crawl/{id} stops a running crawl. Pages already read keep their results and their charge. Pages not yet read are marked cancelled and cost nothing. Cancelling a crawl that already finished returns its status unchanged.

Cost

1 credit per page read. Pages that fail are free. Set limit, maxDepth and path filters to keep the cost predictable, and check the reach first with Map.

Next steps

Map a site

See a crawl’s reach before reading anything.

Read a list of URLs

When you already have the URLs.

Jobs & webhooks

Polling, webhooks and checking deliveries.

API reference

The full schema, with a live playground.