How to build a competitor price tracking dataset

Reliable competitor price tracking starts with a data contract, not a dashboard. Each price must be tied to the correct retail listing or variant, offer conditions, market, collection time, and transformation history before it is safe to compare.

This guide shows how to build that warehouse-ready dataset for a known set of public retail stores. Web Scraper can run source-specific extraction and delivery, while your team retains control of cross-store matching, semantic normalization, acceptance rules, history, alerts, and pricing decisions.

Start with the decision the dataset must support

Define the decision before selecting fields or schedules. A category manager reviewing weekly price position has different freshness and coverage requirements from a pricing team evaluating fast-moving products several times per day.

Write down four requirements:

1. Comparison set: Which products, variants, stores, markets, and currencies are in scope?
2. Decision latency: How old can an observation become before it is no longer useful?
3. Comparison rule: Which price and offer conditions qualify for comparison?
4. Acceptance rule: How complete and trustworthy must a run be before it updates a downstream view?

This model assumes that you already know which retail stores and listings to track. If the job involves search results, rankings, offer discovery, or a changing seller population, use a marketplace monitoring model instead.

Turn the comparison set into an expected-scope table. It should identify every active listing or variant that the system intends to observe, along with store, market, priority, and effective dates. That table provides the denominator for coverage and prevents a shrinking scrape from looking healthy simply because it still returns rows.

Define the competitor price tracking observation

One row should not mean “one product.” A practical observation grain is:

store × source listing or variant × offer context × observed time

The offer context contains the attributes that make two prices meaningfully different, such as market, currency, customer condition, purchase type, or fulfillment terms when relevant. Three sizes must not collapse into one price, and a member price must not silently replace the public price.

Use two different identifiers:

• A stable offer_key identifies the source listing or variant plus its configured offer context across runs.
• A unique observation_id identifies one collected event. If one offer is captured once per run, a uniqueness constraint on run_id + offer_key also prevents retry or pagination duplicates.

The exact contract is customer-owned. It is not an automatic schema produced by a scraping tool.

Build four customer-owned data objects

Keep identity, observations, and consumption views separate so a mapping or normalization correction does not erase source evidence.

Data object	Core contents	Update pattern
Canonical catalog	canonical_product_id, canonical_variant_id, brand, model, size, pack, capacity, and other comparison attributes	Version or effective-date changes
Source-listing map	Store, source listing and variant IDs, URL, canonical IDs, match method, confidence, review state, and effective dates	Versioned mapping changes
Price observations	Raw and parsed price values, offer context, provenance, versions, run ID, timestamp, and quality disposition	Append-only
Current-price view	Latest accepted observation for each canonical identity and offer context	Rebuilt from mappings and observations

The observation fact should make the row’s meaning explicit:

Field group	Example fields	Purpose
Source identity	store_id, source_listing_id, source_variant_id, source_url	Identifies what the retailer exposed
Product evidence	source_sku, gtin, manufacturer_part_number, title_raw, variant_raw	Keeps identifiers separate by scope
Canonical identity	canonical_product_id, canonical_variant_id, mapping_version	Connects comparable items across stores
Offer context	market, currency_code, promotion_type, customer_condition, availability_raw	Explains when and to whom a price applies
Source values	price_text_raw, list_price_raw, promotion_raw, shipping_raw	Preserves what the page displayed
Parsed values	current_price_amount, list_price_amount, availability_normalized	Supports typed analysis without discarding evidence
Provenance	observation_id, observed_at, run_id, extractor_version, schema_version, normalization_version	Makes the row traceable and reproducible
Quality	quality_status, quality_reason	Controls downstream use while retaining exceptions

Adapt these fields to the business decision and each source. Null rules and controlled values should be documented before collection starts.

Separate canonical identity from source listings

A retailer SKU is normally meaningful only within that retailer. URLs can change, and similar titles may describe different pack sizes, colors, capacities, or model generations. Keep source identifiers as evidence, then map them to a customer-owned canonical product and sellable variant.

A valid GTIN is strong matching evidence when available, but retailers may omit it, expose a parent-level identifier, or publish incorrect structured data. Do not assume that store SKUs match across stores or that a similar title proves identity.

Use a controlled evidence hierarchy:

1. Valid global identifiers at the sellable-variant level.
2. Manufacturer part number combined with brand and required variant attributes.
3. Deterministic matching on fields such as size, color, capacity, quantity, and generation.
4. Human review when evidence is incomplete or contradictory.

Record the match method, confidence, review state, and effective dates. Use explicit states such as exact, deterministic, review, and unmatched. An unmatched listing is visible and fixable; a false match can silently drive the wrong pricing decision. When choices require page interaction, configure the sitemap to collect separate product variants and validate unavailable combinations and duplicates.

Model price as an offer with conditions

A bare number is not safely comparable. A page can show a regular price, sale price, crossed-out reference price, member price, coupon, subscription price, range, unit price, or delivered price. Availability, taxes, shipping, and market can also change the interpretation.

Suppose a page displays a public sale price of $79.99, a crossed-out price of $99.99, and a member price of $69.99. Preserve all three source values and label the last one as conditional. A customer-owned comparison rule can select $79.99 for a public-price report without erasing the other evidence.

Keep raw and parsed values side by side. For example, store $24.99 with membership in price_text_raw even if 24.99 is parsed into a numeric field. Web Scraper Parser can perform source-specific cleanup, but currency conversion, pack-size normalization, tax treatment, and delivered-price logic belong downstream. Each method needs an effective time and version.

Treat availability with the same care. “Out of stock” is an observed source state. A listing missing from one run is only “not observed.” It may indicate a removed listing, incomplete pagination, a changed layout, a blocked page, or an extraction failure. Keep absence as a coverage exception until another check establishes its meaning. The separate guide to collecting product names, prices, and SKUs covers source extraction mechanics.

Preserve provenance and transformation versions

When a value looks wrong, the team should be able to distinguish a source change from a bad mapping, parser error, or revised business rule. Retain the exact source URL, source identifiers, run ID, and an explicit machine-readable observation timestamp, commonly UTC.

Keep extractor, mapping, normalization, and schema versions separate. A sitemap change can alter extraction without changing the warehouse schema. A revised unit-price rule can change a normalized value without changing the original observation. Separate versions make investigation and reprocessing possible without rewriting raw evidence.

Choose cadence from decision latency

“Scrape hourly” is not an operating requirement. Begin with the maximum tolerable age of the data, then work backward through the complete pipeline:

latest viable start = decision deadline - extraction - retries - delivery - validation - matching - refresh buffer

This is a planning equation, not a guarantee. Measure scheduled start, actual start, completion, observation, downstream acceptance, and the age of the newest accepted observation when the decision is made.

Web Scraper Cloud supports daily, interval, and custom cron schedules with a selected time zone. If the previous run is still active, the next scheduled run waits. Schedule frequency therefore does not equal usable freshness. Use the recurring job scheduler only after the sitemap has been tested, and raise frequency only when run duration, source behavior, quality, and operating cost remain acceptable.

Put row checks and run acceptance before trusted history

A completed scrape is not necessarily a correct dataset. A successful response can contain a consent screen, challenge, empty shell, or wrong page type. These incorrect page responses require content-level checks, not just request status.

Validate rows first:

• required source identity, price context, and timestamp are present;
• the page contains expected product markers;
• numeric parsing agrees with retained source text;
• run_id + offer_key is unique for the configured grain; and
• customer-defined price plausibility rules flag anomalies for review rather than silently accepting them.

Then evaluate the run against expected scope:

coverage = distinct expected keys passing required row checks / active expected keys

field completeness = candidate in-scope rows with the required value / candidate in-scope rows

Segment these measures by store, category, or priority. A duplicated listing can inflate record count while an important product is missing, so raw row totals are not a substitute for coverage.

Web Scraper data-quality controls can check minimum record count, failed-page percentage, empty-page percentage, and field population. Those extraction-level signals inform the customer's gate, but do not decide whether a batch is fit for pricing decisions. A quality-check failure also does not by itself change a completed job to failed.

Assign a downstream disposition such as accepted, quarantined, or rejected. Only accepted observations may update the trusted current-price view. Retain quarantined data and reasons for investigation without displacing the last accepted state. Do not invent universal thresholds; calibrate them from business risk and observed source behavior.

Append observations and derive current state

Store observations as append-only evidence and derive current state by selecting the newest accepted observation for each canonical identity and offer context.

Observed at	Source variant	Canonical variant	Price raw	Parsed price	Availability	Quality
2026-09-14T06:05:00Z	A-772-BLK	CAT-00491-BLK	$89.99	89.99	In stock	Accepted
2026-09-15T06:07:00Z	A-772-BLK	CAT-00491-BLK	$79.99	79.99	In stock	Accepted
2026-09-15T06:08:00Z	A-772-2PK	Unmatched	$149.99	149.99	In stock	Quarantined

The first two rows form valid history for one comparable offer. The third remains available for mapping review but does not enter the current comparison. This design supports audits, trend analysis, remapping, and reprocessing while giving operational users a simple current view. Derive material changes and alerts downstream rather than treating every difference as a business event. For the general architecture behind event derivation and history, see incremental scraping and change detection.

Connect extraction to the customer-owned pipeline

Web Scraper's role is source-specific extraction and repeatable delivery from configured, tested sitemaps. A practical handoff is:

1. Build and validate a sitemap for each source layout.
2. Run it on a representative sample, including regular and discounted products, unavailable variants, alternate layouts, pagination, and required interactions.
3. Schedule recurring Cloud jobs after the output contract is stable.
4. Use automatic data export for a completed batch or retrieve it through the API.
5. Apply customer-owned matching, semantic normalization, coverage, and acceptance rules.
6. Append accepted observations, refresh current state, and run comparisons or business alerts.

A completion webhook reports job metadata rather than the dataset or a price change. The consumer retrieves the data through the API and must handle repeated deliveries idempotently.

Web Scraper owns configured extraction, sitemap execution, scheduling, extraction-level controls, source parsing, and delivery. Your system owns cross-store matching, semantic normalization, acceptance, historical and current-state models, comparisons, alert thresholds, and pricing decisions. Test JavaScript and interaction requirements against representative pages; compatibility depends on the source.

Roll out before you scale

1. Define the decision, expected scope, grain, keys, null rules, and acceptance logic.
2. Build the canonical catalog and effective-dated source-listing map.
3. Configure one source and preview and validate the sitemap across representative page states.
4. Run several limited cycles and review false matches, missing expected listings, duplicates, anomalies, queue delays, and quarantine reasons.
5. Assign owners for sitemap maintenance, mapping exceptions, data-quality review, and pricing decisions.
6. Expand one dimension at a time: more products, stores, categories, or frequency.

Review each source's robots.txt, terms, access controls, data rights, and applicable law before scaling. The Robots Exclusion Protocol is not access authorization or a complete legal analysis; obtain specialist advice when needed.

Ready to put the model into production? Define the contract and quality gates first, then use Web Scraper Cloud to schedule and deliver the tested extraction into your customer-owned pipeline.