Batch vs continuous web data collection

Batch and continuous web data collection solve different freshness and completeness problems. Batch collection processes a defined scope and publishes a validated snapshot. Continuous collection keeps a persistent queue running and publishes smaller updates as individual targets become due.

Neither model is universally faster, cheaper, or more reliable. The right choice depends on how fresh the data must be, whether consumers require a complete point-in-time snapshot, and how much operational complexity the pipeline can support.

Batch vs continuous collection at a glance

Decision factor	Batch collection	Continuous collection
Work unit	Defined run or partition	Target, page, or small partition
Publication	After the batch passes validation	After each result or small group passes validation
Freshness	Tied to run frequency and duration	Tied to each target’s priority and queue delay
Completeness	Clear snapshot boundary	Must be measured over rolling windows
Recovery	Rerun the batch or failed partition	Retry or quarantine individual tasks
Infrastructure	Usually simpler	Requires durable queues and target state
Traffic pattern	Concentrated collection window	More evenly distributed
Best fit	Reporting snapshots, catalog exports, and periodic audits	Alerts, priority monitoring, and unevenly changing targets
Main risk	One slow or incomplete batch delays the dataset	A growing queue can make data silently stale

Frequency alone does not determine the model. An hourly full-catalog scrape is still batch collection. A system that continuously selects due targets can still execute ordinary finite scraping jobs behind the scenes.

The more useful distinction is the unit of acceptance: does the pipeline publish one complete dataset, or does it accept and publish smaller observations independently?

Start with the data contract

Choose the collection model only after defining what downstream users need.

Requirement	Question to answer
Freshness	How old may an accepted observation become?
Completeness	Must every expected entity be represented before publication?
Consistency	Must records describe approximately the same point in time?
Scope	Is the target inventory known, or must collection discover new pages?
Change distribution	Do most pages change at similar rates, or is activity concentrated in a small subset?
Failure tolerance	Can partial results be used, or must publication wait for recovery?
Recovery boundary	Should one failed page, site, or template be rerun independently?
Consumer behavior	Do downstream systems expect files, snapshots, events, or current-state upserts?

A daily analytics report may require one complete, accepted snapshot by 08:00. A price-alert service may care more about observing selected products within 30 minutes. Those requirements point toward different architectures even when both collect the same fields from the same websites.

How batch web data collection works

A batch has a defined scope, start, and completion condition. It might cover one sitemap, one retailer, one country, or a set of partitions that together form a complete dataset.

A reliable batch pipeline usually follows this sequence:

1. Create a run manifest describing the intended scope.
2. Collect every eligible page or partition into staging.
3. Record failed, empty, and invalid results separately.
4. Validate coverage, required fields, uniqueness, and schema.
5. Accept or quarantine the batch.
6. Publish the accepted snapshot atomically.
7. Derive current-state and historical views from it.

The manifest should identify the dataset, collection window, expected targets, scraper version, schema version, market context, and partitions included in the run. Without it, downstream consumers cannot tell whether an export represents the entire intended scope or merely everything that happened to finish.

Web Scraper Cloud’s Scheduler supports daily, interval, and custom Cron schedules. If the next scheduled time arrives while the previous job for the same sitemap is still running, the next run waits. Configuring an interval shorter than the normal execution time therefore does not increase the actual collection frequency.

When batch collection works well

Batch collection is a strong fit when:

• downstream reporting needs a complete daily or weekly snapshot;
• the full inventory must be reconciled;
• most targets have similar freshness requirements;
• collection has a clear business cutoff;
• changes are detected by comparing accepted snapshots; or
• operational simplicity is more valuable than low per-record latency.

Examples include daily catalog exports, weekly property-market snapshots, directory audits, and complete datasets prepared for a scheduled analytics process.

Batch collection failure modes

The simplicity of a batch does not make it automatically reliable.

Common problems include:

• one slow partition delaying the entire publication;
• publishing an incomplete run because the job technically finished;
• treating absent records as deleted entities;
• rerunning successful pages because one small section failed;
• mixing records collected under different schemas or market contexts; and
• starting the next batch before the previous result has been validated.

A batch should remain in staging until its quality gate passes. Web Scraper Cloud’s data quality controls can check minimum record count, failed-page percentage, empty-page percentage, and required-field population. A quality failure is separate from technical job status, so finished should not automatically mean accepted.

Large batches should also be divided into recoverable partitions. One parent run can still publish a unified snapshot after its required partitions pass. The broader guide to scaling web scraping from thousands to millions of pages covers partitioning, capacity, and bounded recovery in more detail.

How continuous web data collection works

Continuous collection maintains persistent state about what should be collected next. Instead of waiting for one full inventory to finish, the system releases validated updates as individual targets or small partitions complete.

A typical control loop is:

1. Store every target with its priority, last successful observation, and next eligible collection time.
2. Place due targets into a durable queue.
3. Lease work to an appropriate scraper or job partition.
4. Retrieve and validate the expected page.
5. Commit the observation idempotently.
6. Update the target’s state and next collection time.
7. Emit a downstream update when required.
8. Retry, delay, or quarantine failed work according to its failure class.

Useful target-state fields include:

Field	Purpose
target_id	Stable identifier for the collection target
source	Website or data-source boundary
next_due_at	Earliest time the target should be collected again
last_attempt_at	Most recent execution attempt
last_success_at	Most recent accepted observation
priority	Relative scheduling importance
failure_class	Reason the previous attempt failed
attempt_count	Retry control
scraper_version	Extraction contract used
market_context	Country, language, currency, or session state

Continuous does not necessarily mean real time. Most public pages do not send an event when a price, listing, or policy changes. The collector still has to revisit them through polling, schedules, feeds, or another change signal.

The model becomes continuous because scheduling, validation, and publication remain active, not because one HTTP connection remains open indefinitely.

When continuous collection works well

Continuous collection is useful when:

• a small subset of records needs much lower latency than the full inventory;
• pages have very different change rates;
• new results should trigger alerts or downstream workflows;
• failed targets must recover independently;
• traffic should be distributed across the day; or
• current-state data matters more than a simultaneous complete snapshot.

Examples include monitoring high-priority product prices, watching recently published property listings, refreshing active job vacancies, and keeping frequently requested RAG documents current.

Continuous collection failure modes

Continuous systems trade visible batch failures for less obvious forms of staleness.

Typical risks include:

• the queue grows faster than it can be processed;
• one website or page type consumes disproportionate capacity;
• individual targets stop succeeding without affecting the global success rate;
• duplicate or late results overwrite newer state;
• removed entities remain active because no explicit absence rule exists;
• retries continuously recycle terminal failures; and
• the system loses evidence of overall inventory completeness.

Monitor the oldest due task, not only the number of tasks completed. A pipeline may report healthy throughput while its backlog becomes several hours older each day.

Continuous collection also needs periodic reconciliation. If the system refreshes only known product URLs, it cannot discover new products exposed through category navigation. A stream of correct individual updates can still represent an incomplete market.

Why a hybrid model is usually strongest

Many production web-data pipelines combine batch discovery with continuous targeted refreshes.

Layer	Purpose	Example cadence
Full discovery batch	Rebuild the known inventory and find additions	Daily or weekly
Priority refresh queue	Revisit high-value or volatile targets	Every few minutes or hours
Standard refresh queue	Maintain ordinary freshness	Daily or according to observed change rate
Failure queue	Recover transient errors without blocking unrelated work	Backoff-based
Reconciliation batch	Confirm coverage, removals, and stalled targets	Daily
Publication process	Produce complete snapshots or incremental updates	Consumer-dependent

The full batch establishes what exists and provides a completeness boundary. Continuous queues then allocate more capacity to records whose freshness justifies it.

This model is especially useful when change is uneven. Refreshing 250,000 catalog pages every hour is wasteful if only 10,000 offers need hourly monitoring. Refreshing only those known offers, however, may miss new additions. A daily discovery batch plus targeted hourly refreshes handles both requirements.

This architecture is related to, but distinct from, incremental web scraping and change detection. The collection model determines how work is scheduled and accepted. Incremental logic determines what should be revisited and how a new observation changes trusted state.

Compare the freshness calculations

For a periodic batch, the worst normal freshness delay is approximately:

schedule interval
+ batch execution time
+ validation and publication time

If a page changes immediately after it was collected, the pipeline may not observe the new state until the next batch reaches that page.

Suppose a daily batch begins at 02:00, takes four hours, and requires 30 minutes for validation. A page collected near the start of the run can be almost 28.5 hours old by the time the next accepted snapshot is published.

For continuous collection, freshness is closer to:

time until target becomes due
+ queue delay
+ execution time
+ validation and commit time

This can be much lower for priority targets, but only while the queue has enough sustainable capacity.

A stable continuous system needs productive processing capacity to remain above the rate at which work becomes eligible. Adding workers may raise global capacity, but it does not override responsible per-target request rates, browser constraints, or downstream bottlenecks.

Measure freshness at the record level:

freshness age = current time - last accepted observation time

Do not substitute job start time, configured schedule, or webhook receipt time for the actual observation.

Worked example: monitoring a retail catalog

Consider a retailer-intelligence dataset covering 250,000 product pages. The business needs:

• one complete accepted catalog each morning;
• price updates within one hour for 10,000 priority products;
• discovery of newly listed products within 24 hours; and
• no removal alerts based on incomplete collection.

Pure batch approach

Running all 250,000 pages once per day provides the morning snapshot and discovers new products. It cannot meet the hourly priority-price requirement.

Running the entire catalog hourly would attempt six million page collections per day before retries. It would also create repeated traffic spikes and make it harder to finish one run before the next is due.

Pure continuous approach

A continuously scheduled target queue could keep the 10,000 priority products fresh and spread the remaining workload across the day.

However, known product URLs do not reveal every newly added listing. The system would also need extra logic to determine when the catalog as a whole was sufficiently complete for morning reporting.

Hybrid approach

A more practical design uses:

1. a daily discovery and reconciliation batch across category and product pages;
2. an hourly refresh queue for the 10,000 priority products;
3. a lower-frequency queue for stable long-tail products;
4. a quarantine path for invalid pages and exhausted retries; and
5. a morning snapshot assembled only from accepted observations that meet the reporting contract.

A product missing from one incomplete discovery partition remains not_observed. It is not marked as removed until the relevant scope has been collected successfully and the configured confirmation rule is satisfied.

The result supports both low-latency price monitoring and defensible catalog completeness.

Match storage to the collection model

Keep collection execution separate from business entities.

A useful shared model is:

Target -> Attempt -> Page result -> Entity -> Observation

• The target describes the intended page or collection unit.
• The attempt records each execution and retry.
• The page result records what the request actually returned.
• The entity represents the product, property, company, or document.
• The observation records that entity’s accepted state at a particular time.

Batch datasets also need a snapshot_id or run_id that groups accepted observations. Continuous pipelines need target-level state such as last_success_at, next_due_at, and the version currently published.

Use stable entity keys and append observations rather than creating another entity whenever a value changes. Make commits idempotent so a repeated job, webhook, or queue delivery cannot insert the same observation twice.

A possible task key is:

dataset_id + target_id + collection_window + scraper_version

The precise key depends on whether corrected reprocessing should replace, supersede, or coexist with the earlier result.

Use Web Scraper Cloud as the execution layer

For batch collection, create and test the sitemap in the browser extension, then run it through Web Scraper Cloud with the required driver, proxy, delays, and request interval.

The Scheduler can start recurring jobs, while automatic data export can send the complete dataset after a job finishes. Automatic export is not continuous while the job is running, which makes it a natural fit for accepted batch delivery.

For a hybrid or continuously orchestrated workflow, an external application can maintain target priorities and decide when existing sitemap jobs or partitions should run. The Cloud API starts the configured scraping jobs; it is not an arbitrary URL-in, record-out endpoint.

Use a custom_id to associate each job with the internal partition or workflow. When the job reaches a final status, a completion webhook can notify the downstream system. The handler should return a successful response within ten seconds, queue the scrapingjob_id, retrieve the dataset through the API, and import it idempotently.

Webhooks reduce the delay between job completion and downstream processing, but they do not turn a batch job into a streaming scrape. The distinction between notification transport and collection architecture is covered in webhooks vs polling for scraping job results.

Group targets into sensible partitions rather than creating one job for every page. The right unit should be small enough to retry independently but large enough to avoid excessive orchestration overhead.

Monitor the model you actually operate

Batch and continuous pipelines need different leading indicators.

Batch metrics	Continuous metrics
Last accepted run	Oldest due task
Batch duration and p95 duration	Queue delay by priority
Expected versus processed targets	Percentage of targets within freshness SLA
Accepted versus quarantined partitions	Last successful observation by target
Record and field completeness	Task completion and retry rate
Publication delay	Observation-to-publication delay
Difference from previous accepted snapshot	Backlog growth rate

Both models should also monitor:

• expected-page rate;
• required-field population;
• duplicate and uniqueness violations;
• selector and schema versions;
• retries by failure class;
• browser and proxy usage;
• cost per valid observation;
• source-specific request pressure; and
• current-state age by entity and source.

A completed request is not automatically a valid observation. A page can return 200 OK but no usable data because it contains a challenge, consent screen, regional redirect, or empty JavaScript shell. Validate expected content and identifiers before publishing the result.

A practical decision framework

Choose batch collection when most of these statements are true:

• Consumers require a complete point-in-time snapshot.
• One publication deadline matters more than per-record latency.
• The target inventory must be rediscovered regularly.
• The workload can finish comfortably within its collection window.
• Partial results are not useful.
• The team wants the simplest defensible operating model.

Choose continuous collection when most of these are true:

• Selected records need much lower latency than a full crawl can provide.
• Change frequency varies substantially between targets.
• Results should trigger immediate downstream action.
• Individual failures must recover without delaying unrelated data.
• The system can operate durable queues, target state, and idempotent writes.
• Completeness can be measured independently from publication.

Choose a hybrid model when the dataset needs both inventory coverage and low-latency updates. This is the most common answer for catalog, listing, monitoring, and AI freshness use cases.

Move from batch to continuous gradually

Do not replace a working batch with a permanent queue in one step.

A safer progression is:

1. Establish a reliable, validated batch baseline.
2. Split the workload into independently recoverable partitions.
3. Record target-level observation and failure state.
4. Add priority and next_due_at fields.
5. Move only the highest-value targets into a frequent refresh queue.
6. Keep the full batch for discovery and reconciliation.
7. Compare freshness, completeness, source load, and cost before expanding.
8. Retain a rollback path to the last accepted snapshot.

This approach exposes whether lower latency creates enough value to justify the additional operational surface.

Responsible collection applies to both models

Review source terms, access rules, robots directives, data rights, privacy obligations, and the intended use before operating a recurring collection system.

The Robots Exclusion Protocol standardizes how crawler instructions are interpreted, but it does not provide legal permission or settle whether a particular collection and reuse is appropriate.

Batch traffic can create concentrated load, while continuous traffic can become constant background pressure. Start conservatively, control rates per website, and avoid unnecessary revisits regardless of the architecture.

Batch vs continuous collection checklist

Before selecting a model, confirm that you have:

• defined freshness and completeness requirements separately;
• identified whether consumers need snapshots, updates, or both;
• measured representative page and job durations;
• documented the target inventory and discovery process;
• chosen an independently recoverable partition size;
• separated technical completion from data acceptance;
• made retries and downstream imports idempotent;
• defined absence and removal rules;
• established batch-duration or queue-age alerts;
• retained observation time and collection provenance;
• planned periodic reconciliation for continuous targets; and
• reviewed source limits, terms, and applicable obligations.

Batch collection provides clear boundaries and simpler completeness controls. Continuous collection provides lower latency and more precise allocation of capacity. A hybrid pipeline uses each where it is strongest: batches to prove coverage and continuous refreshes to keep important records current.