How JavaScript-rendered content affects web scraping

A product price is visible in your browser. The page loads normally, the product is in stock and the value is clearly displayed. Yet a scraper requests the same URL and returns an empty price field - or no product record at all.

The scraper may not be looking at the same version of the page.

A website can return an initial HTML document, execute JavaScript, request more data and update the page without another full navigation. It may then change the data again when someone scrolls, clicks a button, selects a variation or opens a tab.

JavaScript-rendered content therefore affects more than page-loading time. It changes where the required data exists, when it becomes available and which actions are needed to produce it. Reliable JavaScript web scraping depends on reaching the correct page state, not simply enabling JavaScript and waiting for the page to “load”.

What is JavaScript-rendered content?

When a browser opens a URL, it receives HTML and references to resources such as stylesheets and scripts. It parses the HTML into the Document Object Model (DOM), executes permitted JavaScript and displays the resulting page.

JavaScript-rendered content is content created, inserted or changed by that client-side execution. For example, a page may return an empty results container and then use a Fetch or XHR request to retrieve product data. JavaScript converts the response into product cards and adds them to the DOM.

Modern websites rarely fit into one rendering category. A page may use:

• Server-side rendering: the server returns HTML that already contains the main content;
• Client-side rendering: JavaScript creates the content in the browser;
• Hydration: the server returns pre-rendered HTML, then JavaScript attaches application behaviour to it;
• Hybrid rendering: different parts of the same page use different methods.

React, for example, defines hydration as attaching the application to HTML previously generated on the server. Initial content may therefore be present in the response while filters, navigation and later changes still depend on JavaScript.

The useful question is not, “Does this website use JavaScript?” It is, “Does JavaScript create or change any data or navigation state that this job requires?”

Initial HTML, rendered DOM and interaction state

A scraper may encounter three relevant representations of the same URL.

Representation	What it contains	Typical extraction method
Initial response	HTML returned by the server	HTTP request and HTML parser
Rendered DOM	Document after scripts and supporting data requests have run	Browser-based scraper
Interaction state	Document after scrolling, clicking or making a selection	Browser automation with state-specific extraction

These representations may contain the same data, partially overlapping data or completely different values.

Consider an e-commerce category page. Its initial response contains the page title, filters and an empty results container. JavaScript then requests the category data and inserts the product cards into the DOM.

The URL has not changed, but the available content has. If JavaScript inserts 24 product cards and scrolling loads the next 24, a raw-HTML scraper sees zero products. A browser that executes JavaScript but does not scroll sees 24. A browser that continues scrolling until no new items appear may reach the full catalogue.

All three clients requested the same URL successfully. They captured different states.

How JavaScript changes scraping behaviour

Content can appear after the document loads

JavaScript frequently requests business data separately from the main document. Product results, prices, availability and reviews may arrive through Fetch, XHR or GraphQL requests. If the scraper extracts after the page shell loads but before a response updates the DOM, the request succeeds while the required data remains absent.

Browser events do not completely solve this timing problem. According to MDN’s documentation for DOMContentLoaded, the event fires after the HTML has been parsed and deferred scripts have executed, but it does not wait for every asynchronous operation or resource. The window load event waits for more resources, yet an application can still make later requests or respond to timers and interactions.

Neither event universally means “the price now exists”.

Content can require an action

Executing JavaScript is not the same as reproducing user behaviour. Content may appear only after:

• scrolling an element into the viewport;
• clicking Load more;
• expanding a specification tab;
• selecting a colour, size or location;
• navigating within a single-page application.

Infinite scroll often uses the browser’s Intersection Observer API to detect when a marker approaches or enters the viewport. Waiting ten seconds at the top of the page will not trigger an event that depends on scrolling.

Returning to the catalogue example, scrolling creates more product records. On a detail page, selecting a shoe size may request new stock and price data. Each variation is a different state, so the scraper must select it, wait for the update and associate the resulting values with that size.

Existing values can change

Dynamic rendering does not always add new elements. It can update elements that already exist.

A .price element might be present as soon as the initial DOM is created but contain a skeleton, an empty string or the default variation’s price. A later response changes it to €79.00. Clicking another variation changes it again.

Waiting only for .price to exist would succeed too early. A better condition might be that the element contains a currency value, a loading class has disappeared or the selected variation matches the displayed availability.

The browser’s MutationObserver interface can observe changes in the DOM tree. The practical implication for scraping is that element presence and data readiness are not equivalent.

Navigation may happen without a new document

A single-page application can change the URL and replace the main content without a conventional full-page navigation. Instead of waiting for a new document, the scraper may need to wait for the expected URL, a route-specific element or a supporting response. Navigating through the application can also establish state that is absent when the destination URL is requested directly.

Content may exist outside the main DOM context

If a visible value cannot be selected, it may be inside an iframe or shadow DOM. An iframe has its own document, while a ShadowRoot begins a separate DOM subtree. These are selector-context problems: a longer delay will not move their elements into the main document.

How to tell whether a page requires JavaScript rendering

Do not assume that every modern website requires a browser. Determine where the fields you need actually exist.

1. Search the initial response

Search the raw response for a distinctive visible value such as a product title, SKU or exact price. If every required record, field and discovery link is present, raw-HTML extraction may be sufficient even when the site uses JavaScript elsewhere. A value embedded inside a script or serialised state proves that the response carries it, although extracting it may require a different method from selecting visible HTML.

2. Compare the response with the live DOM

Compare the server response with the live DOM shown in the browser’s Elements panel. If product cards are absent from the response but present in the DOM, browser execution created or inserted them. A JavaScript-capable scraper is likely required unless another stable and appropriate source is available.

3. Inspect Fetch and XHR traffic

Use the browser’s Network panel to find requests supplying the missing content. Their URL, method, payload, status, response and timing reveal whether the scraper extracted too early or the data request itself failed.

Finding JSON does not make the request a public API. Internal endpoints may depend on cookies, temporary tokens or application state and can change without notice. Prefer a suitable supported API when it provides the required coverage and permitted use.

4. Reproduce required interactions

Check whether data changes after scrolling, clicking, opening tabs or choosing values. Define which interactions belong to the dataset: one record per product requires different states from one record per size and colour combination.

5. Check the page and selector context

Confirm that the required values are inside the main document rather than an iframe or shadow root. If the browser received a consent, login, regional or challenge page instead, rendering is no longer the leading issue. Follow the broader diagnostic process in 200 OK but No Data and check the access-related causes described in Why Websites Block Scrapers before changing selectors or delays.

Raw HTML, browser rendering or a hybrid approach?

The right method is the least complex one that consistently produces every required record and field.

Method	Use when	Main advantage	Main limitation
Raw HTML	Required data and discovery links exist in the response	Faster and less resource-intensive	Misses content created only in the browser
Browser rendering	JavaScript, session state or interaction produces required data	Captures the live DOM and interactive states	Greater execution cost and more failure points
Hybrid	Only particular pages, fields or interactions require rendering	Balances completeness and efficiency	Requires separate handling for different page types

A headless browser runs a browser engine without displaying its interface. It enables JavaScript execution and interaction, but it does not know which state is correct or when extraction should begin. A hybrid workflow avoids rendering every URL simply because one part of the project requires it, although it may require separate scraping stages or jobs.

The objective is not to avoid browsers at all costs. Use browser execution where it changes the completeness or correctness of the result.

Wait for data, not an arbitrary amount of time

Fixed delays are easy to configure but weak as readiness conditions. A delay that works during a fast test may fail when a service slows down, while one long enough for every slow response wastes time on normal pages.

Stronger readiness checks are connected to the required data, such as:

• the expected results container exists;
• the expected number of product cards has appeared;
• a price contains a valid value rather than placeholder text;
• a loading indicator has disappeared;
• the item count stops increasing after scrolling;
• the selected variation matches the displayed data;
• an error, challenge or recognised empty-state marker is absent or present as expected.

Tools such as Playwright use auto-waiting and retryable locators, but scraper-specific readiness still requires knowledge of the target data.

“Network idle” can be useful, but it is not universal proof of completion. Analytics, polling, streaming connections and late requests can keep a page active, while the required content may have been ready much earlier. Conversely, a quiet network does not prove that a necessary button was clicked.

Common JavaScript scraping failures

Symptom	Likely cause	Check next
Browser shows records, but raw HTML returns none	Client-side rendering	Compare live DOM and Fetch/XHR responses
Only the first items are extracted	Infinite scroll or load-more pagination	Scroll/click trigger and item-count growth
Element exists, but its value is empty	Skeleton or asynchronous update	Wait for the value, not only the element
Price changes after selecting a variation	Stateful DOM update	Extract within each variation state
Visible content cannot be selected	iframe or shadow DOM	Selector context
Runs are randomly incomplete	Race condition or variable supporting response	Data-specific readiness and saved failure evidence
Browser displays the wrong page	Consent, login, region or anti-bot response	Final URL, screenshot and page landmarks
Longer delays do not reveal more content	Required interaction or failed request	Page events and Network panel

The same empty result can have several causes. Treating every failure as “slow JavaScript” often produces longer jobs without improving the data.

Performance and reliability trade-offs

Browser-based scraping generally requires more CPU, memory and time than parsing an HTML response. Scripts, supporting services and interactions can fail independently, while cookies, region, experiments and session history can change the page that arrives. This reduces throughput and gives each page more ways to produce an incomplete result.

Check page loading and data extraction separately:

1. Did the correct route load?
2. Did the intended page content arrive rather than a login, consent or challenge page?
3. Did JavaScript and the required interactions produce the target state?
4. Did the selectors return the expected records and values?
5. Did record counts and required-field completion pass validation?

A job can pass the first four checks and still produce a poor dataset. For example, a hypothetical job might return 10,000 product records while prices are missing from 18% of them because extraction sometimes began before the pricing request completed. Validate expected counts, empty-page rates and field-population thresholds rather than relying on job completion alone.

Scraping JavaScript-rendered content with Web Scraper

The Web Scraper browser extension lets you build and preview selectors against the live rendered page. For Cloud jobs, use this sequence:

1. Check whether every required record, field and discovery link exists in the returned HTML.
2. Use the Fast driver when it does. Fast extracts raw HTML without executing page JavaScript and cannot run sitemaps that depend on clicking, scrolling or other state-changing actions.
3. Use the default FullJS driver when JavaScript must create or update the target content.
4. Represent required interactions in the sitemap. Use the Element click selector for controls that reveal or change content, and the Pagination selector for pagination or repeated load-more behaviour.
5. Place action selectors above the selectors that depend on them. For example, an Element click selector that opens a specifications tab must appear above the Text selectors that extract those specifications. Selector delays can provide additional time after page loads and actions; apply them to the relevant step rather than adding a long delay everywhere.
6. Preview selectors in the state where extraction will occur. Test pagination on page two rather than only the initial page, and include products with different layouts or optional fields in test runs.
7. If a Cloud run is incomplete, inspect empty and failed pages, reasons and screenshots before increasing delays or rewriting selectors.
8. Add data-quality checks for plausible record counts, failed or empty page percentages and required-field completion.

Running FullJS when the HTML already contains the required data adds unnecessary work. Choosing Fast when the values depend on JavaScript produces incomplete data. Select the driver according to the page state the job must reach.

JavaScript rendering is a page-state problem

A JavaScript-rendered page is a sequence of states produced by the initial response, scripts, data requests, session conditions and interactions. The target must therefore be defined more precisely than “this URL”. It may mean this URL in a particular region, after a request has completed, with a variation selected and after every load-more action.

Reliable JavaScript web scraping follows five steps:

1. Identify where every required field exists.
2. Determine which rendering and interaction state produces it.
3. Use raw HTML or a browser according to that requirement.
4. Wait for evidence connected to the data rather than a generic load event.
5. Validate the extracted dataset, not merely the page request.

Turning JavaScript on is sometimes necessary. Reaching and verifying the correct state is what makes the result dependable.

Frequently asked questions

Does every modern website require a headless browser to scrape?

No. A website may use JavaScript while returning all required data in its initial HTML. Use a browser only when execution or interaction changes the data you need.

Can JavaScript-rendered content be present in page source?

Yes. Server-rendered pages can include visible content in the original HTML and then use JavaScript to hydrate it. Data may also be embedded in a script or serialised application state.

Why does my scraper return 200 OK but no data?

200 OK confirms HTTP-level success, not that the target data rendered. Compare the response with the live DOM and inspect supporting requests. If the wrong page arrived, diagnose the response before changing rendering settings.

Is waiting for network idle enough?

Not always. Persistent analytics, polling or streaming can prevent true network idleness, while content may require an interaction that causes no request until performed. Prefer a readiness condition tied to the required element, value, response or record count.

Can I scrape the JSON request instead of rendering the page?

Sometimes, but first determine whether the endpoint is supported, stable, permitted and reproducible. Internal requests may depend on temporary state and change without notice. A suitable official API is usually more stable when it exposes the required data under workable terms. See Web Scraping vs APIs for a broader comparison.

Why does infinite scroll return only the first items?

The next request may be triggered only when a marker enters the viewport. Waiting without scrolling will not produce the event. The scraper must scroll through the required range and continue until a defined stopping condition is reached, such as no increase in item count.

Are JavaScript-rendered pages harder to scrape reliably?

They introduce more states, dependencies and timing conditions than complete raw-HTML pages. A browser makes those states accessible; reliability still depends on correct interactions, waiting, selector context and validation.

Build a scraper for JavaScript-rendered pages

Build and test a sitemap directly on the target website with the free Web Scraper browser extension. Use the appropriate click, pagination or scroll-down selector when content requires interaction, then import the sitemap into Web Scraper Cloud to schedule jobs with FullJS browser rendering.