Product assortment monitoring with web scraping

Product assortment monitoring uses recurring web scraping to track which products and variants a retailer presents, where they appear and how the range changes over time. It can reveal new listings, apparent removals, category moves and changes in variant depth across public retail and marketplace pages.

Reliable monitoring requires more than comparing two exports. The workflow needs a defined scope, stable product identities, complete snapshots and rules that distinguish a genuine delisting from an incomplete scrape.

What product assortment monitoring measures

A product assortment is the range a seller presents within a defined market, channel and category. Monitoring it can help retailers, brands and distributors answer questions such as:

• Which products or variants have appeared?
• Which products are no longer visible?
• Has a competitor expanded or reduced a brand's range?
• Did a product move between categories?
• Does one retailer offer more colours, sizes or capacities than another?
• Does the visible range differ by country, store or delivery area?

Assortment, price and availability are related but different signals. Price monitoring follows commercial values attached to known products. Availability monitoring asks whether an observed item can currently be purchased. Assortment monitoring first asks whether the product or variant forms part of the observed range.

A product can remain in the assortment while temporarily out of stock. It can also disappear from a category because navigation, location state or extraction failed. The raw observation should therefore be described before it is interpreted. “Not observed in this snapshot” is evidence. “Delisted by the retailer” is a conclusion that usually needs confirmation.

Web scraping supplies the recurring observations for this analysis. The broader e-commerce web scraping workflow also requires product matching, normalisation, historical storage and comparison outside the collection layer.

Start with a monitoring contract

Define the monitoring contract before building selectors or scheduling jobs. It establishes what each snapshot represents and which comparisons are defensible.

Contract element	Decision to record
Source scope	Retailer domains, marketplaces, sellers and allowed entry points
Market context	Country, language, currency, store, postcode and login state, if applicable
Coverage	Categories, brands, search routes or known product URLs
Entity level	Product group, product, variant, offer or a combination
Required fields	Identifiers, title, brand, category, variant attributes, availability, URL and observation time
Cadence	Hourly, daily, weekly or event-driven collection
Snapshot validity	Minimum coverage, required-field completion and acceptable failure rates
Change policy	Rules for additions, temporary absence and confirmed removal
Delivery	Dataset format, destination and responsible downstream system

Cadence should reflect the decision being supported. Daily collection may suit fast-moving electronics, while a weekly snapshot may be sufficient for an industrial catalogue.

State the exclusions as clearly as the coverage. A category-page monitor cannot claim to include products exposed only through search or recommendations. Publicly accessible pages are the clearest fit. Behind-login catalogues and highly personalised ranges introduce additional access, permission and repeatability questions.

Model categories, products, variants and offers separately

Retail sites often combine several entities on one card or product page:

• A category is a navigational grouping such as “Women > Shoes > Trainers”.
• A product group is a model or parent item whose members vary by defined attributes.
• A variant is a specific configuration, such as a black shoe in size 7.
• An offer describes seller-specific commercial state such as price and availability.

Do not flatten these into one title string. A new colour is a variant addition, not necessarily a new product family. A product appearing in two categories is one entity with two category relationships, not two products. On a marketplace, several sellers may offer the same product, so offer rows must not inflate the product count.

Preserve the source category and breadcrumb exactly as collected, then map them to an internal taxonomy in separate fields. If a retailer renames “Trainers” to “Running shoes”, the raw values help distinguish a navigation change from a commercial assortment change.

The Schema.org ProductGroup model provides a useful conceptual distinction between a shared product and variants that differ by size, colour, material or another defined property. Use one row per purchasable variant when the source exposes variant-level identifiers and states. Otherwise, record the available variant set against the product group and document the lower level of precision.

A typical sitemap starts with repeated records on a category page and follows product links for identifiers or attributes found only on detail pages. Web Scraper's guide to scraping listing and detail pages shows how to preserve that relationship.

Variant values may already exist in the page data or appear only after interaction. Test the initial page before adding clicks. When interaction is required, validate that each intended combination becomes a distinct record and disabled choices do not create false variants. The product variant interaction guide covers that workflow.

Use identifiers that survive presentation changes

Titles, images and URLs can change without the underlying product changing. Use a deliberate identity hierarchy rather than a title alone.

For tracking within one retailer, prefer:

1. retailer variant ID;
2. retailer SKU;
3. retailer product or group ID;
4. a stable canonical URL component; and
5. a documented composite key as a last resort.

For matching across retailers, use manufacturer identifiers when they are genuinely available:

1. a valid GTIN for the precise trade item;
2. brand plus the most specific manufacturer-assigned MPN; and
3. a reviewed combination of brand, model and exact variant attributes.

SKU, GTIN and MPN are not interchangeable. A SKU belongs to a merchant's catalogue, while an MPN is assigned by the manufacturer. Google Merchant Center's product data specification recommends stable product IDs and warns against guessing GTINs or MPNs.

Store identifiers at the correct entity level. A product group may have one group ID while each colour and size combination has a separate variant ID or GTIN. Joining group-level identity to variant-level availability can merge states that should remain distinct.

When reliable identifiers are absent, retain the source record and matching confidence. A composite fingerprint based on normalised brand, model and attributes can propose a match, but ambiguous candidates should enter review rather than silently becoming one product.

Capture the complete intended range

A successful first page is not evidence of complete coverage. Products may extend across numbered pagination, next-page links, load-more controls, infinite scrolling, nested categories or alternate layouts for unavailable items.

Coverage testing should include:

• the first, middle and final pagination states;
• categories with different templates;
• products with and without variants;
• sold-out and promotional products;
• legitimate empty categories;
• pagination stopping conditions; and
• duplicates across pages, categories or sellers.

Use the pagination, load-more and infinite-scroll guide to match the sitemap structure to the site's loading pattern. Search-result pages may help discovery, but ranking and personalisation can make them a weak sole source for assortment measurement. Prefer stable category traversal or another explicitly defined range surface when available.

Count discovered pages and records by category. Large unexplained changes in either count can reveal incomplete navigation before it generates false assortment alerts.

Store immutable snapshots

Treat each accepted run as an immutable observation rather than overwriting yesterday's values. Historical records are what allow later logic to separate a brief collection problem from a lasting range change.

A useful snapshot includes:

• source, seller and market identifiers;
• observation time and job ID;
• sitemap or extraction version;
• requested and final URLs;
• raw and normalised categories;
• product-group and variant identifiers;
• raw and normalised attributes and availability;
• page or extraction outcome; and
• validation status.

Web Scraper Cloud's Scheduler can run a tested sitemap at recurring times. Keep its time zone and collection context explicit.

Compare validated snapshots, not consecutive jobs

The comparison baseline should be the most recent validated snapshot for the same source, market and scope, not automatically the immediately preceding job.

A practical sequence is:

1. Validate the new job's coverage and required fields.
2. Select the previous validated snapshot with matching context.
3. Join records using the chosen product or variant key.
4. Compare presence, category membership, attributes and availability.
5. Assign event states.
6. Publish confirmed changes and retain uncertain cases for review.

Event	Interpretation
added	Identifier is present now and absent from the validated baseline
still_present	Identifier appears in both snapshots
availability_changed	Product remains present but its availability changed
variant_added	A new variant appears within an existing product group
category_changed	Category membership changed while the entity remained present
not_observed	The entity was not captured and the run cannot support a removal decision
removal_candidate	The entity is absent from a valid snapshot but has not met the confirmation rule
confirmed_removed	Absence satisfies the agreed confirmation rule

A removal rule might require absence from two consecutive validated daily snapshots. Another workflow might combine a valid category absence with a recognised product-not-found state on the detail page. There is no universal number of missed runs that proves removal, so define and record the rule in advance.

Put a data-quality gate before change alerts

A completed scraping job is not necessarily a correct snapshot. An HTTP 200 OK response can contain a challenge, consent page, soft error or JavaScript shell instead of the expected product content. The guide to diagnosing 200 responses with no usable data explains why status alone is insufficient.

Validate at three levels:

• Page: expected route, page type, positive content markers and recognised empty states.
• Record: required identifiers, valid entity relationships and absence of error text in product fields.
• Job: plausible records by category, pagination coverage, duplicates, failed pages and field-fill rates.

Web Scraper Cloud provides data-quality controls for minimum record count, maximum failed and empty page percentages, and minimum field completion. Thresholds should reflect each sitemap rather than applying one generic rule to every category.

When validation fails, quarantine the snapshot, keep the previous validated baseline, suppress change alerts, diagnose the failure and rerun. Retries may resolve transient loading problems, but they cannot repair changed selectors, incomplete navigation or the wrong regional state.

Worked example: monitoring an appliance range

Consider a distributor monitoring washing machines across three public retail sites. The monitoring contract specifies one country and location state, model-level counting, daily collection and a shared set of required identifiers and attributes.

Each observation contains the retailer product ID, brand, model number, title, capacity, colour, category breadcrumb, availability, product URL, run ID and timestamp. The workflow discovers products through category pagination, visits detail pages, validates coverage, matches records and compares the current valid snapshot with the previous valid one.

On Monday, one retailer produces 142 models. Tuesday's run produces 118, but two pagination pages fail. Products missing from that result receive not_observed, and the snapshot is quarantined rather than generating 24 removal alerts.

Wednesday's valid run produces 141 models. Twenty-three products return, so they create no addition or removal events. One product remains absent. Its category was collected successfully, and the previously known detail-page URL now shows a recognised not-found state. The product becomes a removal_candidate or confirmed_removed, depending on the agreed rule.

The useful result is not the row-count difference. It is the ability to distinguish collection recovery from a persistent source change.

Keep regional and session state explicit

Assortments can vary by country, language, postcode, currency, store, cookie state and user session. Create a market_id or equivalent context key and compare snapshots only within the same state unless the analysis intentionally measures regional differences.

Network location can change availability, regional catalogues, consent screens and page structure. After changing proxy location or type, confirm that the intended market and selectors still appear. A proxy provides a network path, not a guarantee of access or the correct commercial context.

Where Web Scraper fits

Web Scraper provides the collection and delivery layer:

1. Build and test the sitemap in the browser extension.
2. Capture listing, detail-page and variant fields.
3. Confirm pagination, interactions and regional state on a representative sample.
4. Run the tested sitemap in Web Scraper Cloud.
5. Apply scheduling and data-quality rules.
6. Deliver each accepted snapshot to the downstream comparison workflow.

Choose the Cloud driver according to the sitemap. Fast cannot run workflows that rely on scrolling, Element Click, Website State Setup, click-based pagination or pagination links derived from scripts. Those workflows require FullJS.

The Cloud API launches an existing sitemap; it is not an arbitrary URL-in, dataset-out scraping endpoint. Product matching, taxonomy normalisation, historical comparison, confidence rules and business alerts remain downstream responsibilities.

Turn events into useful reporting

Useful measures include active product groups, active variants per group, additions and confirmed removals by period, category coverage, assortment overlap and unresolved not_observed cases.

Keep group, variant and offer counts separate. Ten colours of one shoe can represent one product group or ten purchasable variants. Multiple marketplace sellers can create many offers without increasing the number of distinct products. Duplicate listings, minor variants and navigation changes can otherwise create the appearance of assortment growth.

Alerts should focus on decisions, such as a validated new model in a priority segment, a persistent reduction in one brand's range or expansion into a previously uncovered category. Include the supporting product record and source evidence so an analyst can verify the event.

Before collecting data, review the site's terms, robots rules, applicable law and organisational policy for the intended use. Robots rules guide crawler behaviour but are not legal permission. Minimise personal data and obtain specialist advice when the project creates material legal uncertainty.

The most useful assortment monitor is not the one that reports the most changes. It is the one that can explain what was observed, under which market conditions, from which validated snapshot and why the change was confirmed.