> ## Documentation Index > Fetch the complete documentation index at: https://docs.hydrafetch.com/llms.txt > Use this file to discover all available pages before exploring further. # Map a site > Quickly enumerate a site's URLs without scraping them. Returns synchronously and costs one credit. ## OpenAPI ````yaml https://api.hydrafetch.com/openapi.json post /v1/web/map openapi: 3.0.0 info: title: Hydrafetch API description: >- Hydrafetch is a web data API for developers and agents. Send a URL and get back clean Markdown, the page's own structured data, schema-shaped JSON, links, or a summary, with the navigation, banners and boilerplate stripped out. Scrape one page, crawl a whole site, run a search, or extract to a schema, all through one API with one response shape. Every call costs one credit a page whatever it took to fetch, and failures are never billed. Point us at a whole site and get every page. Ask a question and get answers with per-field confidence and the passage each value came from. You describe the outcome you want — the pipeline decides how to get it. ## Authentication Every request is authenticated with your API key in the `X-API-Key` header. Keys are scoped to a workspace and carry its credit balance. ## Credits Calls are billed in credits and charged only on success. A standard scrape is one credit; richer formats and the extraction tier cost more. Each response reports what it consumed. ## Conventions All timestamps are UTC ISO 8601. Long-running jobs (crawl, batch) return a job id you poll, or a webhook you register. ## Errors Every failure returns the same shape, whatever the status: ```json { "success": false, "error": { "code": "INSUFFICIENT_CREDITS", "message": "Insufficient credits" }, "meta": { "requestId": "019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c" } } ``` Branch on `error.code`, which is stable. `error.message` is written for a person and may be reworded without notice. Validation failures add `error.details`. Quote `meta.requestId` when asking us about a specific failure. Every operation documents the codes it can return. ## Rate limits Limits are per workspace, not per key, over a 60 second window, and the ceiling comes from your plan. Every response carries the IETF RateLimit header fields so you can self-throttle rather than discovering the limit by hitting it: ```http RateLimit-Policy: "workspace";q=600;w=60 RateLimit: "workspace";r=599;t=42 ``` `q` is the quota, `w` the window in seconds, `r` the requests remaining and `t` the seconds until the window resets. A 429 also carries `Retry-After` in seconds; wait that long rather than retrying immediately. The older `X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers are still sent and mean the same thing. ## Versioning and deprecation The version is in the path: every endpoint on this API lives under `/v1/`. Within a version we only make additive changes — new endpoints, new optional parameters, new fields in a response. Adding a field to a response is not a breaking change, so parse defensively and ignore what you do not recognise. Anything that would break an existing integration ships under a new version path instead. When an endpoint or a version is retired, the responses say so before it stops working: ```http Deprecation: @1780272000 Sunset: Sat, 01 May 2027 00:00:00 GMT Link: ; rel="deprecation" ``` `Deprecation` (RFC 9745) marks when the endpoint became deprecated, `Sunset` (RFC 8594) when it stops responding, and the `deprecation` link relation points at what to move to. Nothing on this API is deprecated today, so you will not see these headers yet. Watch for them rather than for an announcement: the headers are the notice, and the linked changelog carries what to move to. version: '1.0' contact: name: Hydrafetch url: https://hydrafetch.com email: support@hydrafetch.com servers: - url: https://api.hydrafetch.com description: Production security: - apiKey: [] tags: - name: Scrape description: >- Turn one URL into clean, LLM-ready content. Ask for Markdown, HTML, links, or the page’s own structured data, and poll a job id when a fetch runs long. - name: Crawl & batch description: >- Whole sites rather than single pages. Map every URL, crawl with depth and path rules, batch a list you already have, and read the webhook deliveries for either. - name: Search description: >- Search the web and get the ranked results back already scraped, so an agent has content to cite rather than links to fetch. - name: Extract description: >- Pull schema-shaped JSON out of one or many pages, with optional per-field confidence and the source passage behind each value. - name: Brand description: >- Resolve a domain into the company ready to render: logos for light and dark, the palette ranked by how the site uses it, fonts, socials, and the design system behind them. - name: Media description: >- Images and screenshots from a page, with source and alt text, or a rendered capture of the page as it appears. paths: /v1/web/map: post: tags: - Crawl & batch summary: Map a site description: >- Quickly enumerate a site's URLs without scraping them. Returns synchronously and costs one credit. operationId: mapUrls parameters: [] requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/MapRequestDto' responses: '200': description: The discovered URLs. content: application/json: schema: $ref: '#/components/schemas/MapResponseDto' '400': description: The request body or query failed validation. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: VALIDATION_ERROR message: url must be a valid URL meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '401': description: >- The X-API-Key header is missing, malformed, or names a key that no longer exists. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: UNAUTHORIZED message: >- Missing X-API-Key header. Agents: https://hydrafetch.com/auth.md meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '402': description: >- The workspace has no credits left for this request. Nothing was charged and nothing was fetched. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: INSUFFICIENT_CREDITS message: Insufficient credits meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '403': description: The key is valid but the workspace may not make this request. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: WORKSPACE_BANNED message: >- This workspace has been suspended and cannot make API requests. meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '404': description: No such job, or the id belongs to another workspace. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: NOT_FOUND message: Job not found meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '422': description: >- The target site did not respond, so there was nothing to return. This is a fact about that URL rather than a fault on our side, and retrying it will not change the answer. Nothing was charged. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: UPSTREAM_UNREACHABLE message: The origin did not respond to any attempt we made. meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '429': description: >- The workspace exceeded its per-minute request limit. Read the RateLimit headers, or Retry-After on this response, and retry after the window resets. headers: Retry-After: description: >- Seconds to wait before retrying. Prefer this over a fixed backoff. schema: type: integer example: 17 RateLimit: description: Requests remaining (r) and seconds until the window resets (t). schema: type: string example: '"workspace";r=0;t=17' RateLimit-Policy: description: >- The quota (q) for your plan and the window it applies over (w), in seconds. schema: type: string example: '"workspace";q=600;w=60' content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: TOO_MANY_REQUESTS message: Rate limit exceeded for this workspace and plan. meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '500': description: Unexpected server error. Nothing was charged. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: INTERNAL_SERVER_ERROR message: Internal server error meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c '503': description: >- A dependency needed for the requested formats is unavailable. Safe to retry. content: application/json: schema: $ref: '#/components/schemas/WebError' example: success: false error: code: SERVICE_UNAVAILABLE message: The summary and json formats are not available. meta: requestId: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c components: schemas: MapRequestDto: type: object properties: url: type: string example: https://example.com description: The site whose URLs you want to enumerate. Must be http(s). includeLinks: type: boolean example: true description: >- Also include same-site links found on the starting page, not just the site's page list. limit: type: number minimum: 1 maximum: 5000 example: 1000 description: Maximum number of URLs to return. search: type: string maxLength: 200 example: pricing description: Keep only discovered URLs containing this term. sitemap: type: string enum: - skip - include - only example: include description: >- Whether to use the site's published page list: skip it, include it alongside discovered links, or use it only. sitemapInclude: type: string maxLength: 200 example: post description: >- On sitemap-index sites, only descend into sub-sitemaps whose URL contains one of these comma-separated terms (case-insensitive). E.g. "post" to target article sitemaps. sitemapExclude: type: string maxLength: 200 example: tag,category,author description: >- Skip sub-sitemaps whose URL contains any of these comma-separated terms (case-insensitive). Compose with sitemapInclude — e.g. include "post" + exclude "tag,category,author" avoids WordPress taxonomy sitemaps (post_tag-sitemap, …-taxonomies-post_tag-*) that also contain "post". order: type: string enum: - newest - oldest - sitemap default: newest description: >- Sort order by sitemap lastmod. "newest" (default) returns the most recent URLs first — the wanted end of a large archive under the limit. URLs without a lastmod sort last. before: type: string example: '2024-06-01T00:00:00Z' description: >- Cursor for paging a large archive newest→oldest: return only URLs with lastmod at or before this ISO timestamp. Page by passing the oldest lastmod from the previous page; dedupe by url across pages. offset: type: number minimum: 0 maximum: 100000 example: 5000 description: >- Skip the first N results (in the sorted order) then return the next `limit`. Pagination for sources with no lastmod (WordPress-core), where the date cursor can't apply. Page sequentially (0, limit, 2·limit, …) so the sub-sitemap cache keeps deep pages fast. includeSubdomains: type: boolean example: false description: Also include URLs on subdomains of the site. Default off. ignoreQueryParameters: type: boolean example: false description: Treat URLs that differ only by query string as one. Default off. required: - url MapResponseDto: type: object properties: data: $ref: '#/components/schemas/MapResultDto' required: - data WebError: type: object required: - success - error - meta description: >- Every failure on this API returns this shape, whatever the status. Successful responses never carry it. properties: success: type: boolean enum: - false example: false error: $ref: '#/components/schemas/WebErrorBody' meta: $ref: '#/components/schemas/WebErrorMeta' MapResultDto: type: object properties: url: type: string example: https://example.com description: The site you requested. links: description: >- The site's discovered URLs, each with its sitemap lastmod (null when unknown). type: array items: $ref: '#/components/schemas/MapUrlEntryDto' count: type: number example: 2 description: How many URLs were returned. total: type: number example: 9051 description: >- Total URLs in the enumeration this page is a slice of. Page with `offset` until offset+count reaches this — do not infer the end from a short page. truncated: type: boolean example: false description: >- Whether enumeration hit its ceiling or time budget. When true, `total` is a floor rather than the real size of the site. required: - url - links - count - total - truncated WebErrorBody: type: object required: - code - message properties: code: type: string enum: - VALIDATION_ERROR - INVALID_INPUT - UNAUTHORIZED - INSUFFICIENT_CREDITS - FORBIDDEN - WORKSPACE_BANNED - NOT_FOUND - TOO_MANY_REQUESTS - UPSTREAM_UNREACHABLE - SERVICE_UNAVAILABLE - INTERNAL_SERVER_ERROR description: >- Stable, machine-readable reason. Branch on this rather than on the message, which may be reworded. example: INSUFFICIENT_CREDITS message: type: string description: Human-readable explanation. Safe to show a user, not to parse. example: Insufficient credits details: type: object additionalProperties: true description: Optional machine-readable context, present on validation failures. WebErrorMeta: type: object properties: requestId: type: string description: Quote this when contacting support about a specific failure. example: 019e8a3c-9f0b-7c12-88ab-1d2e3f4a5b6c MapUrlEntryDto: type: object properties: url: type: string example: https://example.com/blog/hello lastmod: type: object nullable: true example: '2026-05-01T12:00:00Z' description: >- The from the sitemap, if present. null for links found on the page. required: - url - lastmod securitySchemes: apiKey: type: apiKey in: header name: X-API-Key ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.