Skip to main content
Every call answers in the same request when it can. Crawls and batches cover many pages, so they always run as background jobs: you get an id back, then poll it or let a webhook tell you when it is done.

When a call needs more time

A call waits for its result for up to 90 seconds, or for your timeout if you set a shorter one. Almost every call finishes well inside that and returns 200 with { data }, exactly as you would expect. A few take longer, usually a heavily protected page or a large extraction. Instead of failing, the call answers 202 Accepted with a job id, and the work carries on for up to 5 minutes:
Then poll it until status is completed or failed:
string
running, completed or failed.
object
Once completed: exactly the body the call would have returned had it answered in time, { data } included.
object
Once failed: the code, message and HTTP status the call would have answered with, plus any details.
A job is charged once, when it completes, and a failed job costs nothing, the same as a call that answered straight away. Results are kept for an hour after the job ends.
The SDKs and the MCP server handle this for you. An SDK method that gets a 202 polls the job and returns the result as if the call had answered directly. An MCP tool that runs out of time returns the job id, and the job_status tool fetches the result.
To skip the wait entirely, send async: true on a scrape. The call answers 202 with the job id at once.

Crawl and batch

A crawl finds and reads the pages of a site. A batch reads a list of URLs you give it. Both always run in the background. Starting one answers at once with 201:
Then follow it in either of two ways. You can also use both.
Ask for the job’s status to see progress and each finished page:
Batches use GET /v1/web/batch/{id}. Both answer in the same shape:
status goes from running to completed, or ends as failed or cancelled. Each entry in pages has its own status and, once read, its data. Results stay available for 90 days after the job last changes. After that the status call answers 404.

Monitors

Monitors send monitor.change for each change and monitor.check.completed when a check ends, signed the same way as below.

Verifying a webhook

Your webhook URL is public, so anyone who finds it can send to it. Set webhook.secret and we sign every call, so you can reject anything that did not come from us. Each signed call carries these headers: To check a call, compute the HMAC over the raw request body exactly as received. Parsing the JSON and turning it back into text can change the bytes, and then it will never match.
Compare signatures with a constant-time function such as hmac.compare_digest or crypto.timingSafeEqual. A plain == can leak the secret through timing. Always check the timestamp too: without it, someone who captures a real call can send it to you again later.
The timestamp is signed together with the body, so nobody can resend a real call under a new timestamp. If you set no secret, calls are still sent, just unsigned.

Did my webhook arrive?

We keep the history of every call we tried for a job: its status, how many attempts we made and what your endpoint answered last.
Batches use GET /v1/web/batch/{id}/deliveries.
object[]
A failed delivery with a 5xx responseStatus usually means your endpoint was down. A 4xx means we reached you and your endpoint refused the call, so check your signature code before assuming the payload is wrong.

Next: Errors

Every error code, what it means and what to do.