When a call needs more time
A call waits for its result for up to 90 seconds, or for yourtimeout if you set a shorter one. Almost every call finishes well inside that and returns 200 with { data }, exactly as you would expect.
A few take longer, usually a heavily protected page or a large extraction. Instead of failing, the call answers 202 Accepted with a job id, and the work carries on for up to 5 minutes:
status is completed or failed:
Once
completed: exactly the body the call would have returned had it answered in time, { data } included.Once
failed: the code, message and HTTP status the call would have answered with, plus any details.async: true on a scrape. The call answers 202 with the job id at once.
Crawl and batch
A crawl finds and reads the pages of a site. A batch reads a list of URLs you give it. Both always run in the background. Starting one answers at once with201:
- Poll
- Webhook
Ask for the job’s status to see progress and each finished page:Batches use
GET /v1/web/batch/{id}. Both answer in the same shape:status goes from running to completed, or ends as failed or cancelled. Each entry in pages has its own status and, once read, its data. Results stay available for 90 days after the job last changes. After that the status call answers 404.Monitors
Monitors sendmonitor.change for each change and monitor.check.completed when a check ends, signed the same way as below.
Verifying a webhook
Your webhook URL is public, so anyone who finds it can send to it. Setwebhook.secret and we sign every call, so you can reject anything that did not come from us.
Each signed call carries these headers:
To check a call, compute the HMAC over the raw request body exactly as received. Parsing the JSON and turning it back into text can change the bytes, and then it will never match.
Did my webhook arrive?
We keep the history of every call we tried for a job: its status, how many attempts we made and what your endpoint answered last.GET /v1/web/batch/{id}/deliveries.
A failed delivery with a 5xx responseStatus usually means your endpoint was down. A 4xx means we reached you and your endpoint refused the call, so check your signature code before assuming the payload is wrong.
Next: Errors
Every error code, what it means and what to do.