Build a price history dataset from scheduled scrapes

A scheduled scraper gives you repeated price snapshots. It does not create a reliable price history by itself. To build one, you need to preserve when each price was observed, connect observations to a stable product or offer, record gaps honestly and prevent failed or duplicate imports from becoming market data.

This guide shows how to turn recurring scrapes of a known set of public product pages into an auditable time-series dataset. Web Scraper Cloud handles tested extraction, scheduling and delivery, while your downstream system owns historical storage, acceptance rules and analysis.

Define what one price belongs to

Before scheduling anything, decide what one tracked entity represents. Common choices are:

• one product;
• one product variant, such as a particular colour and size; or
• one seller offer for a product in a specific market.

The choice affects both identity and price meaning. If two sizes have different SKUs or prices, they need separate histories. If several sellers appear on one product page, combining them produces a series whose subject can change between runs.

A useful target record might include:

Field	Purpose
entity_key	Stable downstream identifier for the tracked product, variant or offer
source_product_id	Permanent identifier published by the source, when available
sku	Retailer-specific stock-keeping unit
variant	Size, colour or another option that affects price
seller_id	Seller identity when one page contains several offers
market	Country, region or storefront
source_url	Page from which the observation was collected
expected_currency	Currency expected for the target

Prefer a permanent product, variant or offer ID from the source. A documented composite key, such as source domain plus SKU and variant ID, is the next-best option. Use a canonicalised URL only when no stronger identifier exists. Product titles are poor primary identifiers because they can change without the underlying offer changing.

For a fuller guide to creating correctly aligned source records, see how to collect product names, prices and SKUs.

This article assumes that you already know which public pages or offers to track. Cross-store product matching, seller discovery and changing marketplace populations require a separate matching and discovery workflow.

Build three logical data objects

Keep target identity, run status and target-level results separate. This prevents a job-level failure from becoming a product-level event.

Data object	Core fields	Purpose
tracking_target	entity_key, source identifiers, variant, seller, market, URL, expected currency, active status	Defines what each scheduled run is expected to observe
scrape_run	Internal run_id, scrapingjob_id, sitemap ID, schedule slot, actual times, technical status, quality status, counts and processing versions	Records what happened to the collection job
target_run_result	run_id, entity_key, observation status, raw and parsed prices, currency, availability, source URL and observed_at	Records what one run established about one expected target

Use an observation_status such as observed, not_observed, invalid or unknown. Keep it separate from availability, which describes a commercial state such as in_stock or out_of_stock only when the intended page was successfully observed.

An application-owned schema_version, extraction_version and normalisation_version make later investigation and reprocessing possible. Do not assume that every version or timestamp you need will be supplied as a ready-made export column.

Store observation time, not just import time

A price chart should place a value at the time it was observed on the source page, not when a webhook arrived or a warehouse finished importing the file.

Keep these timestamps separate when available:

Timestamp	Meaning
scheduled_for	When the run was intended to occur
observed_at	When the source value was collected
finished_at	When the scraping job completed
imported_at	When the downstream system processed the result

Web Scraper Cloud's Parser can add the scrape time to the output. Configure its time zone and date format explicitly. Parser preview covers up to 100 scraped records, so also validate a downloaded sample when source values vary materially.

Use an unambiguous format such as RFC 3339, normally expressed in UTC for storage and exchange. Store the scheduler or business time-zone identifier separately because a UTC calendar day may not match the market day used by a local pricing team.

The distinction matters when a job is delayed. Web Scraper's Scheduler waits if the next scheduled time arrives while the previous job for that sitemap is still running. A configured hourly schedule therefore does not guarantee evenly spaced hourly observations.

Use actual observation times in calculations. Never infer them from the configured interval.

Preserve raw and normalised prices

A page can show a current selling price, crossed-out price, member price, coupon price, unit price or range. Keep values with different meanings in separate fields.

Field	Example
current_price_raw	€1.299,00
current_price_amount	1299.00
currency	EUR
original_price_raw	€1.499,00
original_price_amount	1499.00
promotion_text	Save €200
availability	in_stock
observed_at	2026-09-17T08:14:31Z

Keep the displayed value after parsing. It provides evidence when a normalisation rule misreads a decimal separator, instalment amount, price range or promotional label.

Use an exact decimal database type or integer minor units with the correct rules for each currency. An inexact binary floating-point value should not be the authoritative amount for price comparisons. A missing or unparseable price should normally remain null with a validation status, not become zero.

Also preserve the conditions that change what a number means, such as membership, tax treatment, unit quantity, shipping inclusion or subscription period. Two numeric values are comparable only when they describe the same commercial unit.

Use dense observations as the evidence layer

There are two common ways to retain recurring price data.

Dense observations

Store one target-level result for every expected entity in every accepted run, including unchanged prices and explicit non-observation states.

This model can prove that a price was checked, reveal the real sampling cadence, measure coverage and distinguish an unchanged price from a missed observation. Its trade-off is greater storage use.

Change-only history

Write a history row only when the accepted price or availability changes.

This is compact and convenient for notifications, but it loses collection evidence. Without a separate coverage record, no new row could mean either “checked and unchanged” or “not observed”.

For a known set of targets, dense validated results are the safer source of truth. Derive a compact price-change view later. If scale requires change-only storage, retain separate run coverage or reliable last_seen_at state for every entity.

This differs from a broader incremental collection system, which may optimise which pages are revisited or republished. Those patterns are covered in incremental web scraping and change detection.

Distinguish unchanged, missing and out of stock

A trustworthy price history must not collapse different kinds of absence into one null price.

Situation	Meaning	Recommended representation
Price observed and unchanged	A valid run saw the same price again	New dense result with the same amount
Page explicitly says out of stock	A valid page exposed a stock state	observation_status = observed, availability = out_of_stock
Expected entity was absent from an otherwise accepted run	No trusted record was collected for that target	observation_status = not_observed
Page was blocked, malformed or could not be validated	The result cannot support a product conclusion	observation_status = invalid or unknown
Entire job failed or its dataset was rejected	No trusted snapshot exists for that run	Failed or rejected scrape_run
No job occurred for an expected slot	Collection did not run	Schedule gap, not a price observation

Out of stock is an observed business state. Not observed is a statement about collection coverage.

A page returning 200 OK can still contain a challenge, consent screen, generic error or JavaScript shell. Before classifying a product as unavailable, confirm that the intended page and product identifier were present. The 200 OK but no data diagnostic guide covers these failure modes in detail.

Do not fill source-data gaps with the last known price. A reporting layer may forward-fill values for a specific analysis, but it should label them as imputed rather than observed.

Worked example: three scheduled runs

Suppose two offers are scheduled every four hours:

Run	Scheduled	Returned	Quality
R101	09:00	2 of 2	Accepted
R102	13:00	1 of 2	Quarantined
R103	17:00	2 of 2	Accepted

In the trusted history, offer A is observed at $49.99 in R101 and $44.99 in R103. Offer B is observed at $79.00 in both accepted runs. The incomplete middle batch remains available for investigation but does not update trusted views.

The dataset supports three conclusions: offer A fell between the two accepted observations; offer B was unchanged across those observations; and the 13:00 state is unknown. It does not claim that offer B was out of stock at 13:00 or that either price remained unchanged throughout the gap.

Build the recurring collection layer

Create and test the sitemap in the Web Scraper browser extension before moving it to Cloud. Include standard prices, promotions, unavailable products, variants and any regional page state that affects the result.

Once the output is correct:

1. Import or synchronise the sitemap with Web Scraper Cloud.
2. Test the same driver, proxy, request interval and page-load delay intended for scheduled jobs. Interaction-heavy workflows that use scrolling or clicks require FullJS rather than Fast.
3. Add the collection timestamp through Parser.
4. Configure scheduled scraping at a cadence the job can sustain.
5. Configure collection-level quality rules.
6. Deliver the completed batch through automatic data export, or use a webhook and the API for a controlled import.

Automatic export sends the complete dataset after a job finishes. It is a delivery mechanism, not a historical database. Your downstream system must append, validate and version accepted results.

Web Scraper Cloud executes the tested sitemap with the configured driver and delivers its output. Stable cross-run identity, historical storage, price comparisons and business decisions remain downstream responsibilities.

Import completed jobs idempotently

A Web Scraper completion webhook contains job metadata such as scrapingjob_id, not the scraped dataset. Use that identifier with the API to retrieve the result.

A resilient import flow is:

1. Receive and validate the webhook metadata.
2. Return a successful response promptly.
3. Queue the scrapingjob_id.
4. Download the completed dataset into staging.
5. Run job-level and target-level validation.
6. Commit accepted results atomically.
7. Refresh derived views.

Keep long downloads and imports outside the webhook request. The endpoint is expected to acknowledge within ten seconds.

Webhook deliveries can repeat after a timeout, unsuccessful response or further processing through Continue. Importing the same data twice must not create duplicate observations. A possible uniqueness key is:

(scrapingjob_id, entity_key, processing_version)

The exact key depends on the reprocessing policy. If the same raw job can be processed again under a corrected schema, retain an import-attempt log and identify each processing version explicitly. Content hashes and a superseded state are one defensible way to distinguish an unchanged retry from a revised candidate batch, but they are downstream design choices rather than Web Scraper features.

Validate and version before accepting history

A technically finished job may still contain an incomplete or incorrect dataset.

Web Scraper Cloud data quality control can monitor minimum record count, failed-page percentage, empty-page percentage and required-field population. A data-quality failure is separate from the technical job status, so do not treat finished as sufficient evidence for acceptance.

Add downstream checks that understand the price-history contract:

• every expected result has a valid entity_key and observation status;
• each entity appears no more than once per intended grain, run and processing version;
• parsed prices use an exact numeric representation and retain the source text;
• currency and market context match the intended target;
• product identifiers agree with the expected page;
• coverage, duplicates, nulls and parse failures remain within calibrated limits; and
• abrupt distribution changes are reviewed before publication.

Quarantine a suspicious run instead of letting it replace the current view. Keep the last accepted observation active until the issue is resolved.

Version the downstream schema, extraction contract and normalisation logic. A queued scheduled job uses the sitemap version captured when the job was created, and later edits do not change that queued job. Your own version fields should make this provenance explicit without assuming the export contains a ready-made version identifier.

Derive current, daily, change and coverage views

Keep trusted target-run results append-only, then create views for common uses.

• Current price: Select the latest trusted observation for each entity_key, ordered by observed_at, not webhook receipt or import time.
• Daily price: Group by the documented business time zone and choose a deliberate rule, such as the final trusted observation of the day or the daily minimum and maximum.
• Price changes: Compare consecutive trusted observations for the same entity and currency.
• Coverage: Summarise expected targets, valid observations, not-observed targets, invalid targets and rejected runs.

A compact change view can use LAG() while leaving the dense evidence layer untouched:

SELECT *
FROM (
    SELECT
        entity_key,
        observed_at,
        currency,
        price_amount,
        LAG(price_amount) OVER (
            PARTITION BY entity_key, currency
            ORDER BY observed_at
        ) AS previous_price
    FROM trusted_price_observations
) history
WHERE previous_price IS NULL
   OR price_amount <> previous_price;

An unchanged dense observation belongs in the source history even though it produces no change event.

Choose cadence from the decision you need to make

A faster configured schedule is useful only when it produces valid observations frequently enough to support a decision.

Choose cadence from acceptable staleness, observed price volatility, normal job duration, campaign or repricing deadlines, collection cost and target-site constraints. High-priority products may justify more frequent checks than a stable long-tail catalogue.

Track both intended and actual cadence. When a job takes longer than its configured interval, subsequent runs wait, so shortening the interval does not necessarily improve real sampling frequency.

Keep regional and variant state consistent across runs so that a different currency, location or selected option does not appear as a price movement. Review target-site terms, robots rules, rate limits, privacy obligations and access boundaries before scaling. Robots rules express requested crawler behaviour, but they are not access authorisation or a substitute for applicable legal review.

Price history rollout checklist

Before using the dataset for reporting or automated decisions, confirm that you have:

• defined the tracked entity and stable entity_key;
• separated targets, runs and target-level results
• stored observation time separately from schedule, completion and import times
• retained raw price evidence beside exact parsed values
• represented unchanged, not observed, invalid and out-of-stock states distinctly
• made webhook and dataset imports idempotent
• quarantined suspicious batches before they update trusted views
• derived current, daily, change and coverage views from accepted history

Start with one validated baseline, then move the tested sitemap into Web Scraper Cloud for recurring execution and delivery. For the broader commercial workflow: