Skip to main content
Some pages come back perfect. Others come back thin, cut short or behind a wall. If your pipeline cannot tell them apart, bad pages end up next to good ones. So every scrape carries a quality object, and your code can decide page by page whether to use the result, retry it or set it aside.
All three are worked out from the page itself, without an LLM. The same page always scores the same, and the check adds no cost or delay.

confidence

A number from 0 to 1. We measure the page’s content in more than one independent way and compare the results. When they agree on a good amount of content, confidence is high. When the page gives almost nothing, or the measurements disagree, confidence drops.

complete

true when the result holds most of the content the page has. false when it looks thin or cut short compared with the page. If you asked for the whole page with onlyMainContent: false, the result is always complete, because nothing was left out.

blocked

true when the page seemed to be behind a challenge or a bot wall and no real content came through. A blocked result always has low confidence and complete: false. A blocked result costs nothing and is never cached, so asking again later fetches the page again rather than repeating the bad copy. When we cannot read a page at all, the call fails with 422 and UPSTREAM_UNREACHABLE instead. See Errors.

Using it in a pipeline

Next: Untrusted content

Why page content is data, not instructions.