POST /v1/web/scrape reads one URL and returns it the way a model wants to read it: clean Markdown, the page’s own structured data, its links, a summary or JSON in your schema. You choose the formats and how the page is fetched. We handle the rest, including pages that need JavaScript to show their content.
When to use it
- You have a URL and want its content now.
- You want more than text: the page’s structured data, its links or a summary.
- You need control over the fetch: wait for slow content, read as if from another country, or send your own headers.
Example
metadata and usage are always there.
Options
What to return
The URL to read. Must be http or https. A bare domain such as
example.com is read over https.Any of
markdown, html, rawHtml, links, structured, summary, json and brand. See Formats for what each returns and costs.For the
json format: a schema (JSON Schema for the shape you want), a prompt (what to look for), or both. One is required when you ask for json.Keep headings, lists and tables, even at the cost of some prose. Good for marketing and service pages, where the headings are the content. Left off, we aim to capture every word, which suits articles but can flatten listing pages into plain text. If
metadata.structure comes back plain and you wanted structure, retry with this on.Keep links inline in the Markdown. By default links are kept where the page’s text relies on them.
true keeps them everywhere; false gives the densest prose.Filtering the page
Remove cookie and consent banners. Set
false to keep them. A banner the site adds with scripts after the page loads is kept only when the page is rendered, so pair it with render: true.Close newsletter and promotion popups when the page is rendered. Set
false to keep them.Always read the page in a browser, as a visitor sees it once its scripts have run. By default a page is read in a browser only when its HTML is not enough. Browser reads count toward how many your plan runs at once; past that you get a
429, so retry with backoff.How the page is fetched
A CSS selector, such as
.price or #results. The page is read in a browser once an element matching it exists. If it never appears within the time budget, you get what the page showed by then, with quality.complete set to false and a warning saying the element never appeared. Up to 500 characters.Read the page as a phone would: a phone-sized screen, touch and a mobile browser. Use it for sites that show phones different content. The page is read in a browser.
How long to wait for the page before answering with a job id instead, in milliseconds, from 1000 to 100000. The wait is at most 90 seconds either way. See Waiting for a result.
Read the page as if from another place.
country is a two-letter code: "US" (United States) or "DE" (Germany). Any other country is refused with a 400 that lists the supported codes, and nothing is charged. Use it for sites that change prices, currency or content by where the reader is. languages lists preferred languages, most preferred first, such as ["en-US", "en"].GET or POST. Use POST for addresses that only answer a POST, such as JSON and GraphQL APIs.Caching
The oldest copy you will accept, in milliseconds. By default 2 days for a page and 1 hour for a feed.
0 always fetches fresh. At most 7 days. See Caching.Answer only from a copy. With no copy young enough, return
404 instead of fetching.Waiting
Answer at once with a job id instead of waiting for the page. Poll
GET /v1/web/jobs/{id}.Response fields
Shortcuts for one format
These take the same options as/v1/web/scrape and return the same response, with the format already chosen:
Waiting for a result
A scrape waits for the page and returns it, for up to 90 seconds, or yourtimeout if it is shorter. Almost every page is back well inside that.
A page that needs longer, usually a heavily protected one, does not fail. The call answers 202 with a job id, and the read carries on for up to 5 minutes:
status is completed, when result holds exactly the response above, or failed, when error says why:
async: true and you get the job id at once. The SDKs poll for you, so a method call simply returns the page. See Jobs & webhooks.
Cost
1 credit per page read, plus 1 forsummary, 4 for json, and 5 for brand when a brand is found. A page we cannot read costs nothing. See Credits.
Next steps
Formats
Everything a scrape can return.
Crawl a whole site
Find and read every page as one job.
Read a list of URLs
Give us the URLs and get them all back.
API reference
The full schema, with a live playground.