Skip to main content
We keep a recent copy of the pages we read. When you scrape a URL we read recently, we can answer from that copy instead of fetching the page again. The answer comes back faster, and it costs the same as a fresh read. By default a scrape accepts a copy up to 2 days old for a page and 1 hour old for a feed. Three options change that.

maxAge

The oldest copy you will accept, in milliseconds. If our newest copy is older, or we have none, the page is fetched fresh.
number
3600000 accepts any copy from the last hour. 0 always fetches fresh. Leave it out for the default. The most you can set is 7 days, 604800000.
Pick it by how often the page changes. A price or a status page wants a short window or 0. A documentation page can take the default.

cacheOnly

Answer only from a copy, and never fetch. If there is no copy young enough, you get 404 instead.
Use it for backfills and replays, where you want what we already have and nothing new.
With cacheOnly, a missing copy is a 404 and no page is fetched. Handle that status in your code.

storeInCache

Whether this read is kept as a copy for later calls. On by default. Turn it off for one-off or sensitive pages you do not want kept.
boolean
default:"true"
Keep this read so later calls within their maxAge can be answered from it.

Telling a copy from a fresh read

A response from a copy says so in two places:
  • data.cached is true.
  • data.usage.freshness is cache. A live read says fresh.
Blocked and very thin results are never kept, so a retry always fetches the page again rather than repeating a bad copy.

Other endpoints

  • extract, images, screenshot and styleguide take maxAge the same way, with 0 for fresh.
  • In a crawl, a batch or a search with scrapeResults, put maxAge inside scrapeOptions to apply it to every page.
  • News reuses a recent answer to the same search for a short time: 5 minutes for the hour window, 15 for day and 30 for week or month. Pass maxAge: 0 to search fresh.

Next: Jobs & webhooks

Waiting for a result, background jobs and completion webhooks.