Scaling web scraping from thousands to millions of pages

A 5 per cent duplicate rate creates 50,000 unnecessary page loads across one million targets. If 10 per cent of those targets fail and every failure receives two retries, the system can generate another 200,000 attempts without collecting a single additional page.

A scraper that handles a few thousand pages does not become a million-page pipeline by multiplying workers. Duplicate URLs consume capacity, retries create traffic spikes, one slow website blocks unrelated work, and a selector change can corrupt an entire run before anyone notices.

Scaling means controlling coverage, throughput, target load, recovery, data quality, freshness and cost. The principles apply to custom stacks and managed execution platforms.

In short

To scale web scraping reliably:

1. Distinguish unique URLs, attempts, page loads, records, entities and observations.
2. Measure a representative pilot before estimating capacity.
3. Separate discovery, retrieval and extraction, with durable work state.
4. Partition work and control request rate per target and globally.
5. Use raw HTML where possible and browsers only for required page states.
6. Classify and bound retries, using stable keys and idempotent writes.
7. Validate coverage, expected pages and data before publication.
8. Measure system health, data health, freshness and cost per valid result.
9. Scale through gated stages with tested recovery and reruns.

One million pages is not a workload specification

Consider two projects.

The first collects one million independent product pages distributed across thousands of websites. The required data is present in the initial HTML, each page takes less than a second to process and the collection runs once.

The second collects one million pages from a single website every day. Each page requires JavaScript, a variant selection and a residential proxy. The target must be accessed conservatively, and every observation must be ready before a downstream morning process begins.

Both contain one million pages but require different completion windows, target pressure, infrastructure and cost. Define the workload precisely before choosing the architecture.

Work units

Workload dimensions

These units are not interchangeable. A listing can produce hundreds of records, one target can generate several attempts, and repeated observations may be valid history. Web Scraper Cloud page credits, for example, count page loads rather than records.

Define an operational contract: collect eligible product pages within 24 hours, populate required identifiers and isolate failures for a bounded rerun. This is more useful than saying the system should "handle a million URLs".

Measure the real workload before scaling it

Estimate capacity from a representative pilot, not one convenient page. Include every important template, slow and JavaScript-dependent pages, pagination endings, regional variations, interaction workflows and known failure states.

Record median and p95 processing time; an average can hide a long browser-page tail. Also measure attempts per target, expected-page rate, required-field population, duplicate rate, transferred bytes, browser share and cost per valid unique record.

Start with simple capacity calculations:

attempts = unique targets x attempt multiplier
attempts per second = attempts / deadline in seconds
average active workers = attempts per second x average attempt duration
provisioned workers = average active workers / target utilisation

Measure the attempt multiplier in the representative pilot by dividing total pilot attempts by unique pilot targets, then apply that ratio to the planned workload. For example, 10,500 attempts across 10,000 unique pilot targets gives a planning multiplier of 1.05. Effective concurrency is what remains productive after delays, per-target limits, browser capacity and downstream bottlenecks, not merely the configured worker count.

For one million attempts, ideal completion time changes quickly with effective throughput:

These figures exclude target limits, backoff, exports and validation.

Suppose a job must process 1,000,000 unique pages within 24 hours. Retrying adds 5 per cent to the attempt volume, the average attempt occupies a worker for three seconds, and the target utilisation is 70 per cent:

1,000,000 x 1.05 = 1,050,000 attempts
1,050,000 / 86,400 = 12.15 attempts per second
12.15 x 3 seconds = 36.45 average active workers
36.45 / 0.70 = 52.07 provisioned workers

Round up to about 53 worker slots as a theoretical starting point. This is not permission to send 12 requests per second to one website; target policy may set a much lower rate.

Load-test the queue, workers, database and export path, with headroom for retries, restarts and downstream delays. Google SRE's guidance on cascading failures explains why overload behaviour must be tested. Full utilisation leaves no recovery margin.

Scale through gated rollouts

Increase scale in stages, with explicit acceptance criteria.

At each gate, compare processed URLs with intended inventory. Stop if throughput rises while expected-page rate or completeness deteriorates, or unit cost rises.

Build a persistent, partitioned URL frontier

At scale, discovery can run ahead of extraction, pagination can create cycles, and a worker failure can lose both the current page and newly found links.

Persist both pending work and the seen set outside worker memory. Marc Najork's overview of web crawler architecture explains why large crawlers partition durable frontiers.

For each target, retain at least:

• the normalised URL and stable canonical key;
• the source page or discovery method;
• the website, template, dataset and partition;
• priority and earliest eligible processing time;
• driver, geography, session and proxy policy;
• state, attempt history and latest failure class;
• scraper and schema versions;
• last successful observation or collection window.

Partition by website, template, driver, geography, freshness or dataset. Each partition should be independently executable, retryable and validatable. A numeric offset alone is unreliable when discovery is dynamic or retries finish out of order.

Set explicit crawl boundaries for allowed hosts, path prefixes, pagination depth and infinite-scroll expansion. Without them, discovery can wander into faceted filter combinations, calendar pages and other effectively unbounded spaces that inflate the frontier without increasing useful coverage.

A practical durable-queue flow is:

1. Discovery normalises and deduplicates URLs.
2. Eligible tasks enter a durable queue with an execution policy.
3. A worker leases a task, retrieves the page and extracts records.
4. Validation accepts, retries or quarantines the result.
5. The worker commits and acknowledges it; expired leases return to the queue.

This flow also shows why the operational distinction between crawling, scraping and data extraction matters at scale: each stage needs its own state and checks.

Normalise before deduplicating

Equivalent resources can have different hostname case, default ports, fragments, paths, encoding or parameters. RFC 3986 covers methods such as lowercasing the scheme and hostname, removing dot segments and normalising percent encoding.

Apply only verified target-specific rules afterwards. Query parameters may encode pagination, variants, language, currency, location or identifiers, so do not remove them indiscriminately.

Use more than one discovery source

Combine XML sitemaps with navigation, feeds, known identifiers, previous collections, supplied inventories or suitable official APIs.

The Sitemaps protocol allows 50,000 URLs or 50 MB uncompressed per sitemap and supports sitemap indexes. Sitemaps can omit pages, retain expired URLs or provide unreliable modification dates, so compare them with navigation and known inventory.

Control request rate, concurrency and session policy

Global concurrency protects shared compute, browsers, connections, proxies and storage. Per-target concurrency prevents one website from consuming the fleet. Request delay controls how frequently requests begin and is distinct from both.

Low concurrency can still create bursts when fast failures are immediately replaced, while browser workers may remain active but spend most of their time waiting.

Apply global capacity alongside per-target budgets using website, template, driver, geography and freshness tags. One failing target should not occupy every worker. Spread launches and retries so the fleet does not reproduce one traffic spike.

Proxy policy belongs inside the target budget. More addresses do not repair duplicate discovery, unlimited retries or broken selectors. Choose routes for geography, compatibility and session continuity.

Keep stateful flows consistent: rotating a route can contradict cookies, location or selected state. Independent pages may use a different policy from multi-step navigation. Compare target-route combinations by expected-page rate and cost per validated record.

Proxy management for web scraping covers these trade-offs. A proxy pool, browser fleet or platform cannot guarantee access to every target; restrictions can depend on accounts, behaviour, sessions, geography, access controls or terms.

Respect rate-limit signals

429 Too Many Requests indicates that the server is applying rate limiting and may include Retry-After, as defined by RFC 6585. 503 Service Unavailable can indicate temporary overload or maintenance and can also include Retry-After, according to RFC 9110.

Honour usable delays and reduce pressure on the affected target. Treat 429 as feedback, not as a reason to rotate IP and immediately repeat the same load. The absence of 429 does not prove the rate is acceptable; latency, resets, timeouts, challenges or degraded content may warn earlier.

Use the cheapest execution path that produces the correct page

Use a browser when JavaScript, background requests, interactions or session state must create the required page. Volume alone is not a reason.

Prefer raw HTML when the response contains the records. For browser pages, wait for a condition tied to the data rather than one fixed delay.

Web Scraper Cloud drivers include Fast for raw-HTML extraction and FullJS for JavaScript execution. One website can need both: a category may require FullJS scrolling while server-rendered product pages use Fast. How JavaScript-rendered content affects web scraping explains how to identify the required state.

Classify templates before execution. A universal browser fallback can double attempts, hide selector failures and turn terminal errors into expensive browser errors. Keep one only when tests show better expected-page and valid-record yield.

In a custom pipeline, test conditional requests. An ETag sent with If-None-Match can produce 304 Not Modified, reducing transfer and parsing. Validate this across required locations and page states.

Budget retries and make repeated execution safe

A retry consumes time, bandwidth, proxy and execution capacity. Classify the failure first.

Cap attempts and elapsed time. AWS guidance on timeouts, retries, backoff and jitter warns that retries amplify overload. Measure recovery by attempt number; an attempt that rarely succeeds may cost more than it returns.

Move persistent failures to a reproducible quarantine or dead-letter queue instead of circulating or hiding them.

Many queues use at-least-once delivery, so tasks can reappear. Amazon SQS documents this behaviour, but every such queue requires idempotent consumers.

Use a stable task key when committing results:

dataset_id + canonical_url + collection_window + page_type

Two workers processing the same task should update or match one result, not create a duplicate.

Model data for updates and recovery

Separate execution history from business data:

Target -> Attempt -> Page result -> Entity -> Observation

The target defines intended work; attempts record executions; the page result stores what arrived; the entity represents the real item; and observations store its state over time. This preserves retry history without duplicating entities.

For e-commerce, separate stable products, SKU-level variants, seller offers and time-stamped observations. Upsert entities by key and append observations so a new price does not create another product.

Commit incrementally by target, window or page type. Keep task state, records, schema versions and failure artefacts separate from published data so one broken template can be rerun alone.

Give each run a manifest with intended scope, dataset and scraper versions, timing and outcome counts. Consumers can then distinguish complete runs, partial reruns and schema transitions.

Retain only diagnostic evidence needed for investigation, exclude secrets and unnecessary sensitive data, and expire obsolete artefacts.

Validate coverage, retrieval and extraction separately

A completed request set can still produce a bad dataset. Separate three responsibilities:

Coverage checks compare intended and processed URLs, pagination and known, new or disappeared entities. Retrieval checks capture status, final URL, template markers, challenge or consent state, region, currency and attempt history. A 200 OK response does not prove the expected page arrived; see diagnosing 200 OK responses with no data.

Validate within them at increasing levels of aggregation:

Use positive controls that every production version must process, plus negative controls for empty, removed and blocked states. Sample across websites, templates, regions and later pagination positions.

Amazon's Deequ research shows declarative constraints and anomaly detection at billion-record scale. Automated checks still need representative source comparison because structural validity does not prove semantic correctness.

Version schema changes. If a repair changes a field's meaning or type, migrate or rerun the partition instead of merging incompatible output.

Measure system health, data health and cost

A scraper can be operationally healthy while its data is wrong. Monitor both views separately.

A 99 per cent page success rate still means 10,000 failures in a million-page run, possibly concentrated in one critical target. Break metrics down by website, template, driver, route and failure class.

Alert on trends and stalls. Growing queue age, collapsing field-fill rates or fewer discovered detail pages can reveal problems early.

Cost per request can reward fast but invalid results. Use:

total cost =
  execution + browser + proxy + transfer + storage + processing
  + retry waste + operational labour

cost per valid unique record = total cost / valid unique records

Track this with expected-page rate, valid-record yield, retry amplification, completeness, freshness and repair time. For recurring work, calculate cost per observation meeting its freshness target. Deduplication, correct driver routing and partial reruns often save more than a lower nominal request price.

Do not refresh every page equally

Recurring collection divides capacity among discovery, high-priority refreshes, failure recovery and lower-priority coverage.

Research on incremental crawling distinguishes rebuilding from selectively refreshing. Revisit frequency can reflect importance, freshness, observed change rate, time since success, previous failure and verified sitemap signals.

Frequently changing offers may need several daily observations, while stable specifications need far fewer. One schedule wastes capacity.

A practical path from thousands to millions

These boundaries are indicative; complexity, distribution and deadlines matter more than volume alone.

Add these controls when replay or manual inspection becomes expensive. Early stages establish the model and validation rules that larger runs repeat.

For a daily dataset of one million product pages, listing jobs discover URLs. The frontier normalises, deduplicates and partitions them by retailer and template. Server-rendered products use Fast; interactive listings use FullJS.

Each retailer gets its own rate, concurrency, session and proxy policy. Results are committed by retailer and window, then checked for identifiers, page type, price state, variants and coverage. A changed template quarantines only its partition. Publication waits for all required partitions and quality thresholds.

In Web Scraper Cloud, this maps to separate sitemaps or jobs, suitable parallel-task capacity, Fast or FullJS selection and targeted reruns.

Responsible operation still applies at scale

Higher capacity increases potential load and consequences. Prefer a suitable official API under workable terms, and review access rules, target terms, data type, intended use and jurisdictions before production.

The Robots Exclusion Protocol in RFC 9309 standardises robots.txt interpretation. Evaluate it during discovery. Compliance does not grant access permission or settle whether collection and reuse are legally appropriate.

Public visibility is not a complete test. Consider copyright and database rights, personal-data processing, retention and redistribution. Restricted access, personal data or substantial copying deserves legal review.

Even without a published limit, begin conservatively, observe responses and avoid unnecessary load. More workers and routes make central enforcement more important.

Scaling checklist

Before a much larger run, confirm:

• Are work units clearly distinguished?
• Has a representative pilot measured median and p95 duration, retry amplification and valid-record yield?
• Does capacity include recovery headroom?
• Are global concurrency, per-target concurrency and request delay controlled separately?
• Can discovery resume, with normalisation before deduplication?
• Are driver, route and session policies assigned by target and template?
• Are retries classified, delayed, jittered, capped and quarantined when exhausted?
• Is repeated task processing safe?
• Can one failing target be isolated?
• Can the system recognise the wrong page behind 200 OK and validate required fields?
• Can failed partitions run again independently?
• Are system health, data health, freshness and unit cost measured separately?
• Are acceptance gates, stop conditions and rollback procedures active?
• Does project review cover access rules, terms, data rights and personal data?
• Has recovery been load-tested?

Adding workers magnifies any missing controls.

Scaling is controlled throughput, not maximum throughput

The objective is to complete the workload on time while preserving coverage, correct pages and data, target-aware rates, recoverability, freshness and predictable cost.

A pipeline may require an inventory, scheduler, worker fleet, browsers, proxies, retries, monitoring, validation, exports and alerts. Teams can build these layers, use a platform or combine managed collection with custom processing. Scraping libraries versus web scraping platforms and build versus buy remain separate decisions from workload definition.

The scalable system is the one that grows without losing control of what is scheduled, retrieved, retried, accepted and published.

Frequently asked questions

How much concurrency is needed to scrape one million pages?

Use the completion window, attempt volume, processing time and safe per-target rate measured in a pilot. Add recovery headroom and check the result against target limits.

Does scraping one million pages require a distributed system?

Not necessarily. One machine can process large raw-HTML workloads. Browser rendering, strict deadlines, stateful flows, isolation or high availability may justify distribution.

Should one million pages run as one job?

Usually not. Partition by target, template, collection window or another recoverable boundary. A parent run can still publish one final manifest.

Should every page be loaded in a browser at scale?

No. Use raw HTML when it contains the data and a browser when JavaScript or interaction creates the required state. Classify templates before execution.

How can duplicate records be prevented when workers retry tasks?

Use stable task and entity keys, idempotent writes, uniqueness constraints and versioned observations. Store attempts separately from entities.

What is the most important large-scale scraping metric?

No single metric is sufficient. Monitor coverage, expected-page rate, valid unique records, retry amplification, completeness, freshness and unit cost.

Scale web scraping without building the execution layer

Web Scraper Cloud provides managed execution, Fast and FullJS drivers, parallel tasks, scheduling, API control, retries, inspection, proxy configuration, exports and integrations. Your application still owns entity modelling, business validation and publication.

A platform cannot guarantee access or remove the need to validate the resulting dataset. Test a representative workflow before estimating full-scale capacity.