200 OK but no data: Diagnosing incorrect page responses

What 200 OK actually confirms

RFC 9110 defines 200 OK as meaning that the request succeeded. For a GET request, the response content represents the target resource.

That says nothing about the fields your pipeline requires. A valid 200 might contain the intended page, a login or challenge screen, a JavaScript shell, a soft 404, a legitimate empty result or content that selectors no longer match.

It helps to treat scraping success as a series of gates:

A 200 response passes the first gate. Production data collection requires all six.

Compare the raw response, rendered DOM and extracted data

When a scraper returns no data, determine which version of the page contains the expected information.

Consider a category page that should contain 24 product cards. It returns 200, but raw-HTML extraction produces zero rows. Its source contains little more than a root element and scripts; in a browser, the cards appear after a JSON request completes. The selectors may be reasonable, but extraction is looking before the data exists. A JavaScript-capable driver and page-specific readiness check are the appropriate fix.

If the rendered browser instead shows a login or challenge page, the same zero-row symptom has a different cause. Inspect the artefacts before choosing the remedy.

Symptom-to-cause checklist

Why 200 OK can still return no usable data

The request ended on a different page

Browsers and many HTTP clients follow redirects automatically. The final response can therefore be 200 even when the original URL ended on a homepage, sign-in screen, consent page, location selector or canonical URL for a different entity.

Record the final URL and redirect chain. Benign canonical redirects are common, so the important test is whether the final host, route and entity identifier still describe the intended page.

Repeated content is another useful signal. If 500 unrelated product URLs all return the same final URL, title and body fingerprint, the site has probably not developed an unexpected enthusiasm for identical products.

The site returned a soft 404

A soft 404 is an error page returned with a successful status. The body may say "not found" or "no longer available" while the response remains 200. Google's crawler guidance similarly identifies this pattern.

Do not confuse a soft 404 with a soft block:

• A soft 404 says the resource is missing.
• A soft block serves an alternative page because of the current client, session or request pattern.

Test a deliberately impossible URL on the same site and compare its title, heading, canonical URL, structural markers and body hash with the suspect response. If the pages are indistinguishable, the status code is not a useful success test.

A challenge or soft block replaced the target content

Security systems do not have to return a neat 403 or 429. A browser check, CAPTCHA, queue or access page may still return 200. Look for challenge-specific headers or text, HTML where JSON was expected, repeated content and failures correlated with request rate, session or network location.

Cloudflare, for example, documents a cf-mitigated: challenge response header and states that its Challenge Pages use text/html regardless of the requested resource type. That makes header and content-type inspection more reliable than assuming every challenge uses one particular status code.

Once confirmed, investigate request pacing, session behaviour and the network path. Our guides to why websites block scrapers and datacenter versus residential proxies cover those decisions.

Do not begin with "try a residential proxy" simply because a selector returned no rows. Retrying a broken selector from a more expensive IP address still leaves you with a broken selector, now with improved geography.

The useful content loads through JavaScript

Many modern sites return a small HTML shell, then populate it using JavaScript and Fetch, XHR or GraphQL requests. Playwright's navigation guidance notes that modern pages continue working after the load event and waits for target elements to become actionable.

Look for empty root containers in the raw HTML, target elements that appear only in the live DOM and content that requires a request, click or scroll. Use a JavaScript-capable browser when required, then wait for evidence connected to the data, such as a listing container or stable item count. An arbitrary delay can hide a timing problem, but it cannot explain one.

The document loaded, but its data request failed

The browser may run correctly while the request supplying its data fails. The document can return 200 while a supporting endpoint returns an error, lacks required state, produces an unexpected empty collection or is cancelled.

Filter the browser's Network panel for Fetch/XHR requests and compare the URL, method, payload, state, status, content type and body. Chrome DevTools can copy a request as cURL for controlled replay. Remove cookies and authorisation headers before storing or sharing copied commands or HAR files.

Finding JSON or GraphQL in DevTools does not make it a public API. Internal endpoints may use temporary tokens and change without notice. If the source offers a supported interface, compare web scraping and using an API before coupling a production pipeline to it.

The same URL returned the wrong representation

Websites may vary content by language, authentication, currency, location, device, experiment or cached state. A product might exist but show no price in the current region, while a logged-out page retains the same route but hides required data.

Compare the failure with a known-good browser session. Inspect Vary, Content-Language, Set-Cookie, Age and Cache-Control, while remembering that Vary is not a complete inventory of personalisation inputs. Reproduce only the page state required and permitted for the collection.

The page is correct, but the selector is not

If the expected value exists in the rendered DOM, the response is no longer the leading suspect.

Common problems include DOM changes, the wrong extraction property, incorrect hierarchy, iframe or shadow-root context and template variants.

Test the selector on the exact failing page state. Check parent selectors first: if a parent matches zero elements, every child will also produce zero. Then confirm both the selected element count and extracted value. A selector finding 24 wrapper elements does not prove that its child price selector found 24 prices.

Pagination or interaction is incomplete

Sometimes "no data" means no data after the first page, or far fewer records than the source contains. Content may also require a click, scroll or load-more action.

Check whether later URLs redirect to page one, parameters change while the body repeats, required interactions are missing or detail-page selectors sit outside the necessary hierarchy.

Use a wait condition tied to expected content where your tooling supports it. A fixed delay should follow evidence that the data is genuinely late. The first page working proves only that the first page works.

The result is legitimately empty

A zero-record result is not automatically a failure. A search may have no matches, a listing may have been removed or a category may contain no products.

Define stable markers for legitimate emptiness, such as a recognised "No results" component, and confirm that the page otherwise has the expected structure. Treat the response as valid only when that marker is present and the empty result is plausible for the URL. Zero records without the expected content or a recognised empty-state marker should be classified as unknown rather than successful.

A step-by-step diagnostic workflow

1. Define a correct result

Choose a representative URL and define its expected page type, entity ID, required markers and fields, plausible record-count range, legitimate empty state and known error markers.

Without an expected state, you can prove only that bytes arrived.

2. Capture the actual response

Record the requested URL, status, final URL, redirects, content type, size, relevant headers, timestamp and raw body. The cURL write-out variables expose these values without mixing them into the saved response.

Search the body for expected and failure markers:

Do not replace the GET with a HEAD request and assume the result is equivalent. HEAD deliberately omits the body, and servers can handle it differently.

Avoid logging full responses indiscriminately. Pages can contain personal data, account details or embedded tokens. A redacted sample, classification and cryptographic hash are often enough for comparison. Never put passwords, tokens or cookie values in ordinary job logs.

3. Classify the returned body

Classify the response as the intended page, a generic or login page, an access response, a soft 404, an HTML shell, a legitimate empty state, an unexpected media type or malformed content.

Inspect the title, main heading, canonical URL, entity identifiers and content markers. Response size is supporting evidence, not a verdict.

If no class can be established, mark the response as unknown rather than successful.

4. Compare raw HTML with the rendered page

If the target is absent from the raw response, load the page with JavaScript and inspect the live DOM. If the content appears, identify the request, state, click, scroll or delay that produced it. If it remains absent, inspect the screenshot and supporting network requests.

5. Test extraction independently

Once the required content exists in the DOM, check element counts, extraction properties, selector hierarchy and iframe or shadow-root context.

6. Change one variable at a time

Compare raw and browser-rendered responses first. Then vary one relevant factor such as driver, page state, interaction, request interval or network location.

Changing the driver, proxy, delay, cookies and selectors together may produce a successful run, but it will tell you very little about the original fault. That matters when the same problem returns next week wearing a slightly different hat.

7. Reproduce on a representative sample

Test known-good, known-empty and recently failing URLs across relevant page types, including later pagination states. A fix is credible when it explains the failure and holds across that sample. "It ran once after lunch" is an anecdote, not a release criterion.

Stop using status codes as data-quality checks

A scrape can produce thousands of rows and still contain duplicates, challenge text or mostly empty fields. Validate after extraction.

Page-level checks

• final route and entity match the requested page;
• content type matches the expected resource;
• required positive markers exist;
• challenge, login and error markers are absent;
• body size and fingerprint are not abnormal;
• page-level record count falls within a plausible range;
• recognised empty states remain separate from failures;
• unrelated URLs do not share a suspicious body fingerprint.

Positive checks are usually stronger than blocklists. Confirming that a page contains a product grid and the requested product ID proves more than confirming that it does not contain the word captcha.

Record-level checks

• required identifiers are present;
• prices and quantities parse into expected types;
• currency accompanies price;
• requested and extracted entity IDs agree;
• extracted URLs belong to expected domains;
• page-level error text has not entered data fields;
• duplicate rates remain plausible.

Job-level checks

• record count remains within an expected band;
• required-field fill rates exceed defined thresholds;
• failed and empty page percentages remain below limits;
• counts by category, region or page type remain plausible;
• changes are compared with previous successful runs;
• suspicious output is quarantined instead of published automatically.

Suppose a catalogue job normally returns 9,500 to 10,500 rows and populates product_price in at least 97% of them. A new run returns 10,000 rows, but only 1,240 contain a price.

The row count looks healthy. Every document may have returned 200. The 12.4% price fill rate is the signal that should stop 8,760 incomplete records from reaching downstream systems.

Thresholds must reflect the source and use case. A hard minimum of one record is adequate for a demo and impressively unhelpful for a daily catalogue feed. Preserve the source URL, capture time and validation outcome so downstream systems can distinguish trusted records from questionable ones. For a broader treatment of collection freshness, provenance and validation, see building a fresh web data pipeline for RAG.

Diagnosing empty results in Web Scraper Cloud

Start in the browser extension by previewing selectors on the failing page and confirming that expected elements exist in the live DOM.

Web Scraper Cloud separates several outcomes that are easy to collapse into "the scraper failed":

• Failed pages returned a 4xx or 5xx response, or did not load.
• Empty pages loaded successfully, but selectors extracted no data.
• No value pages contained matching elements, but no actual values reached the dataset.

Failed pages point towards loading or access, empty pages towards returned content, rendering or selector matching, and no-value pages towards extraction properties or child-selector logic.

For JavaScript-driven pages, compare the Fast and Full drivers. Fast extracts raw HTML without executing page JavaScript, while Full executes JavaScript before extraction. Full-driver screenshots show what the scraper received.

Use Element Preview and Data Preview to test selectors on the relevant page state. For content loaded after navigation, review Page Load Delay and Request Interval in the scraping configuration documentation. A longer delay will not repair an expired session, challenge page or broken selector.

Once the workflow is stable, configure data-quality controls for minimum record count, failed and empty page percentages, and required-field completion. Retries help with transient failures, but they do not repair changed selectors or incorrect regional pages. Data-quality rules stop those mistakes from becoming production data.

Pre-flight checklist

Before accepting a 200 response and publishing its extracted data, ask:

• Did navigation finish at the expected route and entity?
• Does the body use the expected media type and contain positive page landmarks?
• Is there evidence of a login, consent screen, challenge, soft 404 or routed error?
• Are the records present in the raw response, rendered DOM or a dependent request?
• Did required clicks, scrolling and pagination reach the intended state?
• Do selectors match the saved failing DOM and produce actual values?
• Is a zero-record result supported by a recognised empty-state marker?
• Are record counts and required-field fill rates plausible across a representative sample?

If you cannot identify which check failed, classify the response as unknown rather than successful.

Frequently asked questions

Can a 200 OK response have an empty body?

For a normal GET, 200 is expected to carry content, but an application can still return a zero-length or unusable body. Confirm the method, framing and body. Intentional success without content is normally clearer as 204 No Content.

Why is the data visible in my browser but missing from the HTML response?

The browser may create it by running JavaScript and fetching more data. Compare raw HTML with the live DOM and use a JavaScript-capable driver when required.

Does 200 OK mean the scraper was not blocked?

No. Inspect the final URL, headers, content type, title, body markers and screenshot. A challenge, consent screen or reduced page can still look successful at HTTP level.

Should I increase the page-load delay when no data is returned?

Only when the required element genuinely appears later. Prefer a page-specific readiness condition; a larger fixed delay can remain slow and unreliable.

How can I tell whether the page or the selector is wrong?

Inspect the rendered DOM. If the value exists, test the selector and extraction property, starting with the parent selector. If it is absent, investigate navigation, page state, rendering and dependent requests.

Can an API return 200 and still contain an error?

Yes. Some APIs encode failure in the response body, and GraphQL responses can contain partial data alongside errors. Validate the payload and its required fields, not only the status code.

What should I log for an incorrect page response?

Record the requested and final URLs, redirects, timestamp, status, content type, selected headers, body size, classification, scraper version and a redacted sample or hash. Exclude passwords, tokens, cookies and unnecessary personal data.

Diagnose the layer, not the symptom

When a scraper returns 200 OK but no useful data, begin with the final URL and body. Then compare raw HTML with the rendered DOM, inspect dependent requests, test extraction and validate the dataset.

The important question is not simply, "Did the page return 200?" It is, "What page did we actually receive, and does the extracted data meet the contract our pipeline depends on?"