> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hydrafetch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Jobs & webhooks

> When calls return immediately, when they run as background jobs, and how to get results by polling or webhook.

Some calls finish in the moment; others run long enough to hand back a job you follow up on. Hydrafetch keeps the shape predictable: a synchronous result comes back as `{ data }`, and a job comes back as an id you poll — or a webhook that calls you when it's done.

## Scrape: synchronous by default

A scrape waits for the result and returns it inline. You get `{ data }` with the formats you asked for.

```json theme={null}
{
  "data": {
    "url": "https://example.com",
    "status": 200,
    "markdown": "# Example Domain\n\n..."
  }
}
```

If a scrape runs past the synchronous wait — or if you set `async: true` — it returns a job id instead, with a `201`:

```json theme={null}
{
  "jobId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
  "status": "queued"
}
```

Poll it until it's done:

<CodeGroup>
  ```bash cURL theme={null}
  curl https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43 \
    -H "X-API-Key: hf_your_key_here"
  ```

  ```javascript Node theme={null}
  const res = await fetch(
    "https://api.hydrafetch.com/v1/web/scrape/019f3c09-6fae-740f-9257-10c2b6af7f43",
    { headers: { "X-API-Key": "hf_your_key_here" } },
  );
  const job = await res.json();
  ```
</CodeGroup>

The status response reports the job state and, once finished, the page:

<ResponseField name="status" type="string">
  `waiting`, `active`, `completed`, or `failed`.
</ResponseField>

<ResponseField name="data" type="object">
  The scraped page, present when `status` is `completed`.
</ResponseField>

<ResponseField name="error" type="string">
  Set when `status` is `failed`.
</ResponseField>

<Note>
  Set `async: true` up front when you know a page is heavy — a JavaScript-rendered app, a long settle time — so your request never blocks waiting for it.
</Note>

## Crawl and batch: always async

A crawl (discover and scrape a whole site) and a batch (scrape an explicit list of URLs) always run as background jobs. Starting one returns an id immediately with a `201`:

<CodeGroup>
  ```json Crawl theme={null}
  {
    "crawlId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
    "status": "queued"
  }
  ```

  ```json Batch theme={null}
  {
    "batchId": "019f3c09-6fae-740f-9257-10c2b6af7f43",
    "status": "queued"
  }
  ```
</CodeGroup>

You then track it one of two ways.

<Tabs>
  <Tab title="Poll">
    Request the job's status endpoint for live progress and per-page results:

    ```bash theme={null}
    curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43 \
      -H "X-API-Key: hf_your_key_here"
    ```

    Batches use `GET /v1/web/batch/{id}`. Both return the same status shape:

    ```json theme={null}
    {
      "id": "019f3c09-6fae-740f-9257-10c2b6af7f43",
      "kind": "crawl",
      "status": "running",
      "total": 100,
      "completed": 42,
      "failed": 1,
      "creditsUsed": 42,
      "pages": [
        { "url": "https://example.com/blog/post", "status": "completed", "depth": 1, "data": { } }
      ]
    }
    ```

    `status` moves through `running` to `completed` (or `failed` / `cancelled`). Each entry in `pages` carries its own `status` and, once done, its scraped `data`.
  </Tab>

  <Tab title="Webhook">
    Register a `webhook` when you start the job and Hydrafetch calls you as it progresses — including the full status when it finishes — so you don't have to poll.

    ```bash theme={null}
    curl -X POST https://api.hydrafetch.com/v1/web/crawl \
      -H "X-API-Key: hf_your_key_here" \
      -H "Content-Type: application/json" \
      -d '{
        "url": "https://example.com",
        "limit": 100,
        "webhook": {
          "url": "https://your-app.com/hooks/hydrafetch",
          "events": ["page", "completed"],
          "secret": "whsec_a_long_random_string",
          "headers": { "Authorization": "Bearer your-token" }
        }
      }'
    ```

    Two events fire:

    | Event | When |
    | - | - |
    | `crawl.page` / `batch.page` | Each time a page finishes — carries that page's `status` and scraped `data`. |
    | `crawl.completed` / `batch.completed` | Once, when the job reaches a terminal state — carries the full status. |

    <ParamField body="webhook.url" type="string" required>
      Where to POST events. Must be a **public** http(s) URL — private, loopback, and link-local addresses are rejected with a `400`.
    </ParamField>

    <ParamField body="webhook.events" type="string[]">
      Which events to receive: `page`, `completed`, or both. Defaults to both. Use `["completed"]` if you only want the final result and not a call per page.
    </ParamField>

    <ParamField body="webhook.headers" type="object">
      Extra headers sent with each callback — use them to authenticate the request to your endpoint.
    </ParamField>

    <ParamField body="webhook.secret" type="string">
      A shared secret. When set, every callback is signed so you can verify it really came from Hydrafetch. See [Verifying a webhook](#verifying-a-webhook).
    </ParamField>

    <Note>
      Deliveries are durable. If your endpoint is down or returns a non-2xx, we retry with exponential backoff (up to 6 attempts) rather than dropping the event.
    </Note>

    <Note>
      The webhook and polling aren't exclusive. You can register a webhook and still poll the status endpoint whenever you want.
    </Note>
  </Tab>
</Tabs>

## Verifying a webhook

Your webhook endpoint is a public URL, so anyone who finds it can POST to it. Set a `webhook.secret` and we sign every callback, letting you prove a request came from us and reject anything that didn't.

Each signed delivery carries these headers:

| Header | What it is |
| - | - |
| `X-Hydrafetch-Signature` | `t=<timestamp>,v1=<hmac>` — HMAC-SHA256 of `<timestamp>.<raw body>`, keyed with your secret |
| `X-Hydrafetch-Timestamp` | Unix seconds at send time |
| `X-Hydrafetch-Event` | The event, e.g. `crawl.completed` |
| `X-Hydrafetch-Delivery` | Unique id for this delivery attempt |

To verify, recompute the HMAC over the **raw request body** — not a re-serialized version of the parsed JSON, which may produce different bytes and never match:

```python theme={null}
import hashlib, hmac, time

def verify(raw_body: bytes, signature_header: str, secret: str, tolerance_s: int = 300) -> bool:
    parts = dict(p.split("=", 1) for p in signature_header.split(","))
    timestamp, received = parts["t"], parts["v1"]

    # Reject anything too old — this is what stops a captured call being replayed at you later.
    if abs(time.time() - int(timestamp)) > tolerance_s:
        return False

    expected = hmac.new(
        secret.encode(), f"{timestamp}.".encode() + raw_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, received)  # constant-time
```

<Warning>
  Compare signatures in constant time (`hmac.compare_digest`, `crypto.timingSafeEqual`) — a plain `==` leaks timing information. And always check the timestamp: a signature alone stays valid forever, so without a freshness window a captured delivery can be replayed at you.
</Warning>

The timestamp is signed along with the body, so an attacker cannot take a real delivery and re-send it under a fresh timestamp. If you don't set a secret, callbacks are still sent — just unsigned.

## Checking whether a delivery landed

A webhook that never arrives is the hardest kind of bug to chase from your side, so we keep the delivery history and expose it. Every callback we attempted for a job is listed, with its status, how many attempts we made, and what your endpoint returned last.

```bash theme={null}
curl https://api.hydrafetch.com/v1/web/crawl/019f3c09-6fae-740f-9257-10c2b6af7f43/deliveries \
  -H "X-API-Key: $HYDRAFETCH_API_KEY"
```

Batches use the same shape at `GET /v1/web/batch/{id}/deliveries`.

<ResponseField name="deliveries" type="object[]">
  <Expandable title="WebhookDelivery">
    <ResponseField name="id" type="string">Delivery id.</ResponseField>
    <ResponseField name="event" type="string">The event this delivery carried.</ResponseField>
    <ResponseField name="status" type="string">`pending` while retries remain, `success` once delivered, `failed` once we gave up.</ResponseField>
    <ResponseField name="attempts" type="number">How many delivery attempts have been made.</ResponseField>
    <ResponseField name="responseStatus" type="number | null">The HTTP status your endpoint returned on the last attempt.</ResponseField>
    <ResponseField name="error" type="string | null">Why the last attempt failed, if it did.</ResponseField>
    <ResponseField name="deliveredAt" type="string | null">When the delivery succeeded (ISO 8601).</ResponseField>
    <ResponseField name="createdAt" type="string">When the delivery was first created (ISO 8601).</ResponseField>
  </Expandable>
</ResponseField>

A `failed` row with a `responseStatus` of 5xx usually means your endpoint was down when we called; one with a 4xx means we reached you and you rejected it — check the signature verification above before assuming the payload is wrong.

## Which calls return what

| Call | Response | If you ask for a job instead |
| - | - | - |
| Scrape | `{ data }` inline, or `504` past 90s | `async: true` returns `{ jobId, status }` — poll `GET /v1/web/scrape/{id}` |
| Map, Search, Extract | `{ data }` inline | — |
| Crawl | Always `{ crawlId, status }` | Poll `GET /v1/web/crawl/{id}` or register a webhook |
| Batch | Always `{ batchId, status }` | Poll `GET /v1/web/batch/{id}` or register a webhook |

A single URL is always answered inline unless you opt out. Crawl and batch are jobs because they
span many URLs and can run for minutes; one page cannot, so it does not get a second response shape
you have to write code for.

<Card title="Next: Errors" icon="https://mintcdn.com/hydrafetch/koPVMLXM3S4OTC3p/icons/warning.svg?fit=max&auto=format&n=koPVMLXM3S4OTC3p&q=85&s=060ab47078ad7d4ad1803e44b7d45052" href="/concepts/errors" width="18" height="18" data-path="icons/warning.svg">
  The structured error model and every status you'll see.
</Card>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.