Incremental web scraping and change detection

A page can change without the required data changing. The required data can change while the initial HTML remains identical. A record can disappear from one run without being deleted.

Those cases are easy to collapse into a single question: did the page change? A reliable incremental scraping system asks more precise questions. What should be revisited? Which representation contains the required data? Does the new observation differ from trusted prior state? Is that difference meaningful? Is the run reliable enough to publish an update?

Incremental web scraping is therefore a stateful synchronisation problem, not simply a scheduled scrape with a checksum attached. It uses state from earlier runs to reduce unnecessary collection and apply only validated record changes downstream.

What incremental web scraping actually means

Incremental scraping collects and processes only the data needed to bring a stored dataset up to date.

There are three separate opportunities to avoid repeated work:

1. Collection: Decide which URLs or discovery pages need to be fetched again and which collection method they require.
2. Comparison: Determine which extracted entities are new, changed, unchanged, missing or potentially removed.
3. Downstream processing: Write, notify, index or analyse only the records affected by a confirmed change.

The saving can occur at any of these stages. A provider feed may identify likely updates. A conditional request may avoid transferring an unchanged response body. A response fingerprint may prevent unnecessary rendering or extraction. A record hash may prevent an unchanged entity from being rewritten, re-embedded or published again.

These are different optimisations. A collector often cannot know whether a page changed until it requests the page again. Research into online content change scheduling describes this as incomplete change observability: most sources do not expose a complete remote signal for every meaningful update.

The system should also distinguish an observation from a change event. An observation records what a valid run saw at a particular time. A change event is the conclusion produced after comparing that observation with trusted prior state. Keeping them separate makes retries, audits and uncertain results much easier to handle.

Start with stable entity identity

Change detection is impossible until the pipeline can answer a more basic question: are these two records observations of the same entity?

A source URL is useful, but rarely sufficient on its own. URLs can gain tracking parameters, change slugs, redirect or represent several variants. One product page can also contain multiple offers, sizes or locations that need separate histories.

Choose the record grain first, then define a stable key for that grain.

Prefer a permanent ID supplied by the source. If none is available, use a durable domain identifier such as a SKU or listing ID, followed by a documented composite key built from stable fields. Treat a canonicalised source URL as a fallback, not the default identity strategy.

Keep the observed and final URLs as provenance even when neither is the primary key. If the same entity ID appears under a new URL, classify it as moved or as a URL update. Treating it as one deletion and one unrelated new record creates noisy history and misleading events.

Do not merge records merely because their titles look similar.

Choose the level at which change matters

There is no single universal object called "the page". A target may have an HTTP response, raw HTML, a rendered DOM, data returned by a Fetch or XHR request and several extracted business records. Each layer answers a different question.

Use the cheapest trustworthy signal first, but let the comparison closest to the downstream use decide what the change means.

For example, a product page can rotate an advert, renew a session token and update its copyright year while the product record remains identical. A raw HTML hash will change, but the dataset should not. Conversely, a price loaded after the initial document can change while the application shell remains byte-for-byte identical. In that case, an unchanged HTML hash hides the update that matters.

JavaScript makes the representation especially important. If the required data arrives through Fetch or XHR, the main document's validator says nothing conclusive about that later response. If the workflow depends on the rendered DOM, keep locale, currency, cookies, authentication and interaction state consistent. The difference between the initial response and the rendered page is covered in how JavaScript-rendered content affects web scraping.

Use source signals without trusting them blindly

Before downloading and comparing every page, inspect the update signals the source already exposes. A supported API with cursors, update timestamps or event delivery is usually a better discovery layer than repeatedly crawling every known detail page. Feeds and sitemaps can also help discover or prioritise URLs.

These signals need testing. Google uses sitemap lastmod values when they are consistently and verifiably accurate and represent a significant update. This makes lastmod useful for prioritisation, but not proof that every unmarked page is unchanged. A publisher's definition of a significant update may also differ from the fields your dataset monitors.

HTTP validators provide another useful layer. A client can store an ETag and later send it in If-None-Match, or store Last-Modified and send If-Modified-Since. A server may then return 304 Not Modified instead of sending the representation again, as defined in RFC 9110 and RFC 9111.

Prefer a reliable ETag when both validators are available. Modification dates are limited by timestamp precision and server maintenance, while If-None-Match takes precedence over If-Modified-Since. Even so, the origin controls the validator. A weak ETag can remain stable across changes the origin considers equivalent, and the selected representation may vary by headers, cookies, authorisation or language.

A 304 is therefore a transfer optimisation, not proof that a rendered price or extracted record is correct. Conditional request headers also are not configurable within a Web Scraper Cloud sitemap. Retain application-level record comparison and a slower full reconciliation even when source signals appear reliable.

Compare canonical records, not raw pages

A cryptographic hash can tell you whether its input changed. It cannot tell you whether the input was the right thing to compare.

Raw HTML is noisy. Cookie banners, recommendation blocks, generated element IDs, advertising, timestamps and reordered attributes can change while the target record remains the same. Instead, extract the fields your dataset depends on and build a canonical representation.

A typical normalisation step may:

• trim and collapse irrelevant whitespace;
• apply consistent Unicode normalisation;
• convert prices to a consistent numeric format and currency representation;
• normalise dates and time zones;
• resolve relative URLs and remove tracking parameters;
• sort object keys before serialisation;
• sort collections only when their order has no meaning;
• preserve the distinction between missing, null, an empty string and zero;
• exclude collection metadata such as scraped_at, run IDs, request IDs and session tokens.

Do not over-normalise. Lowercasing every string can hide a meaningful code change. Rounding prices can erase a real update. Sorting a ranked list destroys information when order matters. Normalisation should remove irrelevant variation without discarding business meaning.

The resulting record can be serialised with stable field ordering and hashed. Pass already normalised values into the hash function. For example, represent money as integer minor units or another fixed decimal form so that 12.99, "12.99" and 12.990 do not become different versions of the same price.

If the new hash equals the current hash for the same entity key, the comparable record is unchanged. If it differs, calculate a field-level diff before writing the new version. The hash is an efficient equality test; the stored payload and field-level diff explain what changed.

Keep more than one comparison policy when necessary. A catalogue pipeline may preserve every accepted content change but notify users only when price, stock or delivery changes. This separates source fidelity from business significance.

A production incremental scraping workflow

The following workflow keeps collection, validation and change application separate.

1. Maintain a source and URL inventory

Store every known URL with its canonical form, entity key where available, discovery source, last successful check, last content change and current status. Continue crawling category pages, pagination, sitemaps, feeds or other indexes because a pipeline that checks only known detail URLs cannot discover new entities.

Remove duplicate routes before scheduling them, but retain the originally discovered URL for diagnostics. Define the scope of every discovery surface so the pipeline knows whether a completed run represents a full catalogue, one category, one region or another partial view.

2. Assign refresh policies

Group URLs by volatility, business value, source behaviour, collection cost and the cost of serving stale data. Active offers might be checked hourly, ordinary product details daily and stable documentation weekly.

Recent valid changes can temporarily raise priority, while a long history of unchanged observations can reduce it within an allowed freshness limit. Every class should still have a maximum check interval and a slower reconciliation cadence.

Add jitter so many jobs do not start at exactly the same moment. Respect rate limits and Retry-After, as well as robots rules, target terms, privacy obligations and access boundaries. Do not let failed, blocked or incomplete runs train the scheduler as if the source were unchanged.

3. Collect a comparable representation

Use the least expensive execution method that reliably produces the required fields. Raw HTML is appropriate when the data is present in the response. A JavaScript-capable browser is necessary when the page renders records, applies variants or loads results after interaction.

The comparison must use the same logical page state across runs. A price captured before selecting a variant cannot be compared safely with a price captured after selection. Different locations, currencies or authenticated states may produce valid but incomparable records.

Version the extraction contract. Store an extractor_version, normalisation_version and schema_version with each observation so a selector, sitemap or transformation change is not mistaken for a website-wide update. When the contract changes materially, reprocess the previous snapshot with the new transformation or perform an explicit, auditable re-baseline before normal change detection resumes.

If a request returns 200 OK but the expected records are absent, do not classify the entities as deleted. The response may contain a challenge, consent screen, routed error or JavaScript shell. Diagnose the returned representation first, as explained in 200 OK but no data.

4. Validate the run before applying changes

A completed job is not automatically a valid snapshot. Check the run against expectations before allowing it to update trusted state.

Useful gates include:

• expected discovery-page and pagination coverage;
• minimum record count or a bounded deviation from recent valid runs;
• failed and empty page rates;
• required-field population;
• duplicate-key rate;
• expected page type and positive content markers;
• distribution checks for important values such as price and availability.

Quarantine a run that fails its quality contract. It is safer to retain the last known good state than to publish a large false deletion or replace valid data with challenge pages.

5. Classify each observation

After normalisation, compare each candidate with the trusted current record for the same entity key.

Keep invalid observations separate from this business taxonomy. A record that fails its own schema or quality checks should be quarantined rather than classified as a source change.

6. Store observations, current state and history separately

Begin with a trusted baseline. Run a complete collection, validate its coverage and required fields, assign stable keys and store the first accepted versions. Decide deliberately whether this baseline should emit new events or seed state silently.

Maintain a fast current view and an auditable history. The current table holds the latest accepted payload, hash, status, version and observation timestamps. An append-only history or change log preserves accepted versions with their validity intervals, run IDs and changed fields. Keep the underlying observations so the system can distinguish what was collected from what was eventually published.

Never overwrite the only copy of the old value. History is needed to debug extraction changes, reverse a bad run and distinguish source updates from pipeline defects.

7. Publish changes idempotently

Once the candidate run passes validation, update the current view and emit downstream events. Consumers might refresh a search index, update a warehouse or trigger a price alert. In a retrieval system, only changed documents should proceed to chunking and embedding, while superseded chunks must stop appearing in ordinary retrieval. See building a fresh web data pipeline for RAG for that workflow.

Make every event explicit. Include the entity key, change type, changed fields, old and new hashes, source URL, source timestamp where available, observation time, run ID and schema version.

Use an immutable run ID and an idempotency key for every applied change. Reprocessing the same run should not create duplicate versions or notifications. Update current state and durably queue the event in one transaction or an equivalent recoverable sequence, such as an outbox pattern. Otherwise, a failure between the state update and event publication can lose notifications.

Also protect against out-of-order completion. If an older run finishes after a newer one, compare observation time and version state before allowing it to replace current data.

Detect deletions without corrupting the dataset

Deletion detection is the most dangerous part of incremental scraping because absence has several possible meanings.

A record may be missing because it was removed, but it may also be outside the current crawl scope, hidden behind pagination, temporarily unavailable, returned in another regional or variant state, omitted by a broken selector or lost during a partially failed run.

Use evidence appropriate to the source:

• a verified, repeatable 404 or 410 on the expected route for a previously valid detail page;
• a clear inactive, sold or closed status on the page;
• absence from a source-provided complete inventory;
• absence from multiple successful reconciliations or throughout a defined grace period;
• a verified redirect or canonical move to a replacement entity;
• confirmation through a second discovery path.

A 404 Not Found does not establish whether the condition is temporary or permanent. A genuine 410 Gone is stronger evidence, but the pipeline should still verify that it is not looking at a block, incorrect route or regional response.

Represent removal as a state transition or tombstone rather than immediately erasing the record. Store when it was last seen, when removal was first suspected, what evidence confirmed it and when it became inactive.

For listing-based sources, absence is meaningful only within a complete and valid scope. A global deletion decision should not be based on one partial category, region, filter or paginated view.

Where Web Scraper Cloud fits

Web Scraper Cloud can provide the managed collection layer while application-specific comparison and version policy remain in the downstream system.

A practical setup is:

1. Build a sitemap that extracts stable IDs, canonical URLs, comparable fields and any source timestamps.
2. Choose the Fast driver when the required data is present in raw HTML, or the JavaScript-capable FullJS driver when the page must render or interact first.
3. Use the Scheduler for recurring collection, or start jobs through the API when your own refresh queue decides what should run.
4. Configure data quality control for minimum record counts, failed and empty page percentages and field population. Let the downstream quality gate decide whether the completed run may update trusted state.
5. Use a completion webhook to queue import and comparison after a job finishes.
6. In the downstream pipeline, normalise records, calculate hashes, apply the deletion policy, write accepted versions and notify consumers.

Webhook delivery should be treated as at least once. A notification can be retried, and further processing of the same scraping job can produce another notification. Key the handler by scraping job ID and downstream processing version so repeated callbacks remain safe.

The boundary is important. Web Scraper Cloud handles repeatable navigation, rendering, extraction, scheduling, collection-quality controls and delivery. Your data layer remains responsible for cross-run identity, canonicalisation, comparison, version history, deletion policy and business-event publication.

Monitor the complete incremental pipeline

A healthy scheduler can still feed an unhealthy dataset. Monitor the full path from discovery to published change.

Segment these metrics by source and refresh class. A single average can hide a volatile source that is checked too slowly or a stable source consuming most of the collection budget.

Alert on silence as well as volume. Zero changes may be normal for a stable source, but it may also mean discovery stopped, a selector broke or every candidate was quarantined.

Common incremental scraping mistakes

Choose a hybrid refresh strategy

For most production projects, a hybrid design works best:

• use feeds, source timestamps and change history to prioritise likely updates;
• revisit discovery surfaces so new entities can enter the inventory;
• use HTTP validators where they are accurate and controllable;
• compare canonical record hashes to keep unchanged data out of downstream processing;
• adapt refresh intervals using valid observations, business value and acceptable staleness;
• run slower full reconciliations to catch missed changes and confirm removals.

An incremental system still performs some repeated collection. Its efficiency comes from checking intelligently, comparing reliably and restricting expensive downstream work to confirmed changes.

Frequently asked questions

Does incremental scraping mean downloading only changed pages?

No. Without trustworthy source signals, the scraper may need to request a page to prove it is unchanged. Record hashes can still prevent unnecessary database writes, notifications, indexing and other downstream processing.

Should I use ETags or content hashes?

Use both when appropriate. A reliable ETag can avoid transferring an unchanged representation. A canonical record hash compares the extracted fields your dataset cares about and remains under your control. Neither is a substitute for validating the collected page state.

Should I hash the HTML or the extracted data?

For structured pipelines, hash a deterministic normalised record or an explicit allow-list of business fields. A raw HTML hash can be a cheap screening signal for stable machine-readable responses, but ordinary pages often contain adverts, tokens and layout noise unrelated to the dataset.

How do I detect newly created pages?

Continue checking listing pages, pagination, XML sitemaps, feeds and relevant indexes. Revisiting only known detail URLs cannot reveal new entities.

How do I detect removed records?

Treat absence as a missing observation first. Confirm that the relevant scope was collected completely and validly, then require sufficient evidence such as repeated misses, a grace period, an explicit inactive state or a genuine 410 Gone. Store a tombstone instead of deleting history.

How often should an incremental scraper run?

Base cadence on acceptable staleness, observed volatility, business importance and collection cost. Use refresh classes rather than one global interval, set minimum and maximum intervals, add jitter and retain a slower reconciliation schedule.

Can Web Scraper Cloud perform change detection automatically?

Web Scraper Cloud can schedule or start recurring jobs, collect rendered or raw page data, apply collection-quality controls and notify a downstream system when a job completes. Cross-run entity matching, canonical hashes, field-level diffs, version history and deletion rules should be implemented in the downstream data pipeline.

Automate the collection behind your change-detection pipeline

Build and validate your sitemap locally, then use Web Scraper Cloud to run it on a schedule or through your existing workflow. Start a free trial and connect repeatable web data collection to the comparison, versioning and notification logic your application requires.