Skip to main content
POST /v1/web/scrape reads one URL and returns it the way a model wants to read it: clean Markdown, the page’s own structured data, its links, a summary or JSON in your schema. You choose the formats and how the page is fetched. We handle the rest, including pages that need JavaScript to show their content.

When to use it

  • You have a URL and want its content now.
  • You want more than text: the page’s structured data, its links or a summary.
  • You need control over the fetch: wait for slow content, read as if from another country, or send your own headers.
For a whole site, use Crawl. For a list of URLs, use Batch.

Example

The response holds the formats you asked for. metadata and usage are always there.

Options

What to return

string
required
The URL to read. Must be http or https. A bare domain such as example.com is read over https.
string[]
default:"[\"markdown\"]"
Any of markdown, html, rawHtml, links, structured, summary, json and brand. See Formats for what each returns and costs.
object
For the json format: a schema (JSON Schema for the shape you want), a prompt (what to look for), or both. One is required when you ask for json.
boolean
default:"false"
Keep headings, lists and tables, even at the cost of some prose. Good for marketing and service pages, where the headings are the content. Left off, we aim to capture every word, which suits articles but can flatten listing pages into plain text. If metadata.structure comes back plain and you wanted structure, retry with this on.
Keep links inline in the Markdown. By default links are kept where the page’s text relies on them. true keeps them everywhere; false gives the densest prose.

Filtering the page

boolean
default:"true"
Return only the main content, without menus, headers and footers.
string[]
CSS selectors to keep. When set, only matching elements are used. Up to 50.
string[]
CSS selectors to remove before reading. Up to 50.
boolean
default:"true"
Remove images embedded as base64 text.
boolean
default:"true"
Remove ads and tracking elements.
boolean
default:"true"
Remove cookie and consent banners. Set false to keep them. A banner the site adds with scripts after the page loads is kept only when the page is rendered, so pair it with render: true.
boolean
default:"true"
Close newsletter and promotion popups when the page is rendered. Set false to keep them.
boolean
default:"false"
Always read the page in a browser, as a visitor sees it once its scripts have run. By default a page is read in a browser only when its HTML is not enough. Browser reads count toward how many your plan runs at once; past that you get a 429, so retry with backoff.

How the page is fetched

number
Extra milliseconds to let the page settle before reading it, from 0 to 30000.
string
A CSS selector, such as .price or #results. The page is read in a browser once an element matching it exists. If it never appears within the time budget, you get what the page showed by then, with quality.complete set to false and a warning saying the element never appeared. Up to 500 characters.
boolean
default:"false"
Read the page as a phone would: a phone-sized screen, touch and a mobile browser. Use it for sites that show phones different content. The page is read in a browser.
number
How long to wait for the page before answering with a job id instead, in milliseconds, from 1000 to 100000. The wait is at most 90 seconds either way. See Waiting for a result.
object
Read the page as if from another place. country is a two-letter code: "US" (United States) or "DE" (Germany). Any other country is refused with a 400 that lists the supported codes, and nothing is charged. Use it for sites that change prices, currency or content by where the reader is. languages lists preferred languages, most preferred first, such as ["en-US", "en"].
object
Extra headers to send to the site, such as a cookie that picks a store or a region.
string
default:"GET"
GET or POST. Use POST for addresses that only answer a POST, such as JSON and GraphQL APIs.
string
The request body to send with POST, as written. Set its content type yourself in headers.

Caching

number
The oldest copy you will accept, in milliseconds. By default 2 days for a page and 1 hour for a feed. 0 always fetches fresh. At most 7 days. See Caching.
boolean
default:"false"
Answer only from a copy. With no copy young enough, return 404 instead of fetching.
boolean
default:"true"
Keep this read as a copy for later calls.

Waiting

boolean
default:"false"
Answer at once with a job id instead of waiting for the page. Poll GET /v1/web/jobs/{id}.

Response fields

object

Shortcuts for one format

These take the same options as /v1/web/scrape and return the same response, with the format already chosen:

Waiting for a result

A scrape waits for the page and returns it, for up to 90 seconds, or your timeout if it is shorter. Almost every page is back well inside that. A page that needs longer, usually a heavily protected one, does not fail. The call answers 202 with a job id, and the read carries on for up to 5 minutes:
Poll it until status is completed, when result holds exactly the response above, or failed, when error says why:
To skip the wait, send async: true and you get the job id at once. The SDKs poll for you, so a method call simply returns the page. See Jobs & webhooks.

Cost

1 credit per page read, plus 1 for summary, 4 for json, and 5 for brand when a brand is found. A page we cannot read costs nothing. See Credits.

Next steps

Formats

Everything a scrape can return.

Crawl a whole site

Find and read every page as one job.

Read a list of URLs

Give us the URLs and get them all back.

API reference

The full schema, with a live playground.