How to scrape infinite scroll pages and lazy-loaded content
September 01, 2026
Web Scraper selectors, lazy loading, Web scraping automation, data quality, infinite scroll
To scrape infinite scroll pages reliably, identify the action that reveals new records, reproduce it with the right selector, and stop only when a defined dataset boundary has been reached. Scrolling, clicking Load more and waiting for a field to appear are different workflows, even when the page makes them look similar.
The goal is not simply to move a browser to the bottom. It is to produce a complete, correctly structured dataset that can pass the same checks on every run. That requires a dataset contract, a defensible stopping rule and validation from the first batch to the last.
Diagnose how the page loads records
Keep the technical distinction compact. Infinite scroll is a navigation pattern that requests or reveals another batch when the user approaches a trigger. Lazy loading defers a resource or field until it is needed. A virtualised list displays a moving window of records while reusing or removing DOM nodes outside the viewport.
| What changes after scrolling or clicking? | Likely pattern | Scraping approach |
|---|---|---|
| New record wrappers are added | Infinite scroll | Scroll the repeating Element selector |
| A button adds another batch | Load-more pagination | Use a Pagination selector for the control |
| Existing records remain, but images or fields fill in | Lazy-loaded fields | Render the page and wait for the required field state |
| Visible rows change while the DOM count stays nearly constant | Virtualised list | Track unique IDs throughout the journey and apply the fallback below |
| A Next link or numbered URL opens another page | Conventional pagination | Follow the finite links instead of scrolling |
Run one controlled browser pass before building the sitemap. Count the initial record wrappers, scroll once, wait, and check whether the count rises or existing nodes change. Look for an inner scrollable panel and for a Load more control. Watch Fetch or XHR activity to see whether an action returns records, a cursor or an end marker, but do not assume an observed private request is a stable or permitted API.
Also compare the returned source with the live DOM. If all required records and discovery links are already in the raw HTML, browser interaction may be unnecessary. If JavaScript creates the records or an action changes the page state, use a browser-based workflow. The guide to JavaScript-rendered content and web scraping covers that broader rendering decision.
Define the dataset contract before the selector
Define the row, key and boundary before choosing the trigger and selector. This prevents the page’s interface from becoming the accidental specification for your dataset.
First, state what one output row represents and which fields are required. Choose a stable record key such as a product SKU, listing ID, vacancy ID or canonical detail-page URL. A title is usually too fragile because names can repeat or change. The key exposes duplicated promoted cards, reordering and batches that were requested twice.
Then record the scope:
- the category, query, location, sort order and filters that must remain active;
- the oldest date, maximum record count or natural end state;
- whether promoted and organic records belong together;
- whether card-level fields are sufficient or detail pages are required; and
- which fields must be populated before a row is accepted.
Consider a hypothetical recurring job-listings collection. One row is one vacancy, keyed by vacancy ID, with title, employer, location, posting date and URL required. The run collects listings posted since an illustrative cut-off date and partitions start URLs by location. Those decisions make duplicates, gaps and out-of-scope rows measurable. All numbers used in such a test should be treated as illustrative, not as claims about a particular site.
For recurring work, bounded segments are easier to validate and rerun than one search that continues indefinitely. Technical access also does not settle whether collection and use are permitted under applicable terms, privacy, copyright or other obligations.
Choose the right Web Scraper configuration
Build and test the sitemap in the free browser extension on the real page. Preserve the required filters and sort order in the start URL or reproduce that state explicitly in the sitemap.
Use an Element selector when scrolling appends records
Create an Element selector for the repeated product cards, vacancy rows or property listings and enable Multiple. Add Text, Link, Image or Element Attribute selectors as children so every field is extracted within its record wrapper. Element Preview should highlight complete records, not the entire results container or fragments within cards.
Enable Scroll on that Element selector. The Element selector documentation also describes the Element limit, which is useful when the dataset contract calls for a fixed sample or maximum. The scroll-selector tutorial provides a working sitemap pattern, but its selectors still need to be adapted to the target page.
Use a Pagination selector for Load more
When a control reveals the next batch, model the data-producing event as pagination. Select the button and place the record-extraction selectors beneath the Pagination selector. Start with Auto. If the control is not recognised correctly, use Click multiple times on next/more button, which continues until no new records are scraped.
The Pagination selector documentation explains the available modes, while the Load more tutorial shows the intended selector tree. Use Element Click for an interaction within a record, such as revealing specifications, rather than for list navigation.
Wait only for the state you need
If a record wrapper exists before a required field reaches its final value, use rendering and a bounded delay at the relevant step. Check for the real value rather than a skeleton, placeholder or temporary image attribute. A longer global delay cannot repair a wrong wrapper, blocked response or failed supporting request.
Set stopping rules in a clear hierarchy
An infinite interface still needs a finite collection rule. Apply the strongest available signal first and retain a safety boundary.
- Prefer a terminal state. An end-of-results message, disabled Load more button or disappearance of the control is stronger evidence than a fixed number of interactions.
- Confirm no new unique records. Another scroll or click should add no new stable keys. Page height and loading animations are not record evidence.
- Enforce the business boundary. Stop at the stated date, category edge, record target or other scope limit. An Element limit is appropriate when the requirement itself is a fixed maximum.
- Partition large searches. Use non-overlapping categories, locations, date ranges or price bands so each segment can finish and be rerun independently.
- Treat execution time as a ceiling. Web Scraper Cloud allows up to 10 minutes of data-extraction execution per URL. If an interaction-heavy page reaches the limit, no data is scraped from that page. Design each URL to finish comfortably inside the ceiling rather than using time as proof of completeness.
Virtualised lists require a firm decision. Record an early stable ID, scroll far enough for that row to leave the viewport, and check whether its node still exists or has been reused. If the scraper captures every unique record before recycling, validate the export across the whole journey. If it cannot, the final DOM cannot prove completeness. Do not add more delay. Narrow the query or use a suitable finite, official or otherwise permitted discovery route.
Build locally and prove dataset completeness
Start with the smallest path that produces one correct row, then add fields, scrolling or pagination. Test the second and later batches before running a bounded end-to-end scrape. The site-scraping documentation explains selector trees, Element Preview, Data Preview and timing settings.
Cover boundaries where loading logic is likely to change:
| Test case | What it should prove |
|---|---|
| Short result set contained in one batch | Initial records are extracted without unnecessary interaction |
| Medium set spanning several batches | Stable keys accumulate and earlier records remain represented |
| Large set near the chosen boundary | The business limit and safety controls behave as designed |
| Empty result | No placeholder or interface element becomes a record |
| Final partial batch | The last records are retained even when the batch is not full |
| Slow response or missing optional field | Timing and row acceptance rules do not shift fields between records |
Evaluate the export at three levels.
Navigation completeness: confirm that every in-scope batch was reached. Compare unique IDs with a trustworthy visible total when one exists, or verify the expected end marker. For the hypothetical job-listings run, inspect records on both sides of the illustrative date boundary and compare each location segment with its defined scope.
Record integrity: sample records from the first, middle and final batches. Calculate duplicate rate from the stable key and completion rates for required fields. Check that related values, such as vacancy title and employer, remain attached to the same ID. A plausible row count can still hide later fields being read from the wrong wrapper.
Run repeatability: repeat the bounded test and compare unique counts, keys, missing-field patterns and stopping points. Investigate an unexplained drop, a repeated initial batch or a changed final key before automation. Keep the last accepted export and sitemap version as a baseline.
A successful response is not evidence that the intended dataset arrived. A challenge, login page or JavaScript shell can still return HTTP success. When output is empty or implausible, use the 200 OK but no data diagnostic workflow before changing delays at random.
Move a proven sitemap to Cloud
Move the exact locally validated sitemap to Web Scraper Cloud and run a small remote pilot before scheduling it. Use the Full (FullJS) driver for scrolling, Load more and other state-changing JavaScript interactions. The Fast driver extracts raw HTML without executing JavaScript, so it is unsuitable when the sitemap depends on scrolling or clicks. Fast is a good option only when the required records and discovery links are already present in the response.
Compare the pilot export with the accepted local baseline. In the Cloud job’s Inspect view, distinguish failed pages, empty pages and no-value pages. For Full runs, screenshots can show what the scraper actually received, while job counts and the performance graph can reveal where record production stopped. The Web Scraper Cloud documentation covers drivers, inspection and the per-URL execution ceiling.
For recurring delivery, configure data-quality controls for minimum record count, maximum failed-page percentage, maximum empty-page percentage and minimum field completion. Base thresholds on accepted test runs and business tolerance. Quarantine a failed dataset rather than sending incomplete rows downstream automatically.
Troubleshoot common failures
| Symptom | Likely cause | Check next |
|---|---|---|
| Only the initial batch is extracted | Scroll is disabled, the wrong container moves or Fast is selected | Repeating wrapper, actual scroll panel and Full driver |
| Load more works once | Pagination mode or button selector is wrong | Click-multiple-times mode and changed button class, label or state |
| Scrolling continues but no rows appear | Wrong wrapper, a later container, failed request or virtualisation | Wrapper after batch two, network response and unique keys |
| Records repeat | Promoted cards, shifting sort order or a repeated cursor | Stable ID, active filters and pagination state |
| Later fields are blank or misaligned | Fields settle late, card templates differ or children sit outside the wrapper | Later-batch preview, field readiness and selector hierarchy |
| Local results work but Cloud is empty | Driver, consent, locale or returned page differs | Full screenshot, start URL, filters and Inspect classification |
| The job stops before the visible end | Overlay, rate limit, source cap, changed control or execution ceiling | Screenshot, terminal marker, button state and smaller partitions |
Change one variable at a time and preserve the failing sample. Otherwise, a passing rerun may not reveal which change fixed the problem.
Prefer finite discovery when available
Scrolling should reproduce a necessary interface, not become the default discovery method. Numbered pages, stable URL parameters or published sitemap files are easier to partition, retry and audit. Web Scraper supports ranged start URLs, and its Sitemap.xml selector can discover and filter target pages from published sitemap files.
Prefer an official API or feed when it supplies the required fields on workable terms. Do not treat a private Fetch, XHR or GraphQL request observed in developer tools as a documented public API.
Frequently asked questions
What if the visible total changes while the scraper runs?
Use stable keys and record the query, filters, sort order and run time. For fast-changing sources, a visible total is a useful range check, not a perfect reconciliation target. Smaller date or location segments reduce drift and make overlaps easier to identify.
Should duplicates simply be removed after export?
Deduplicate by stable key for analysis, but investigate why duplicates occurred. Repeated cursors, promoted cards and changing sort order can also conceal omissions, so a clean final count does not prove that every in-scope record was captured.
Build a repeatable infinite-scroll workflow
Start with the free Web Scraper browser extension to define the record wrapper, configure scrolling or Load more and validate the complete export locally. Once the sitemap reaches a proven boundary and passes first, middle and final-batch checks, move it to Cloud with Full rendering, inspection and data-quality controls for repeatable collection.