How to track competitor product launches automatically
September 08, 2026
e-commerce data, product launches, web scraping
Tracking competitor product launches reliably requires more than watching pages for visual changes. The practical method is to collect comparable catalogue snapshots, validate every run and compare stable product identities with accepted history.
A newly observed row is only a candidate. It becomes a useful launch signal after you rule out restocks, new variants, relisted products, regional introductions, catalogue backfills and collection errors.
Decide what counts as a product launch
“Alert me when a competitor adds something new” sounds precise until the first comparison produces dozens of additions.
A catalogue can generate several events that look new without representing a new product launch:
| Event | What happened | Classification |
|---|---|---|
| Net-new product | A previously unseen product family entered the monitored catalogue | Product launch |
| New variant | A known product gained another size, colour or configuration | Variant addition |
| Restock | A known item changed from unavailable to available | Availability change |
| Relist | A previously seen item returned after an absence | Relisted product |
| Regional introduction | A known product appeared in another country or storefront | Market-specific introduction |
| Backfill | Improved coverage exposed an item that was already present | Collection backfill |
| Moved or renamed | The same product ID appeared under a new URL or title | Product update |
| Uncertain | The run was incomplete, blocked or inconsistent | Hold for review |
Define these states before configuring alerts. Otherwise, routine assortment changes enter the same queue as genuine launches, and users stop trusting the feed.
Keep three dates separate:
source_release_date: a release date explicitly provided by the website, when available.first_seen_at: when your system first observed the product.confirmed_at: when the observation passed the rules required for a launch event.
The date first observed is your detection time. It does not prove when the competitor originally launched the product.
Choose the product identity and monitoring scope
Launch detection depends on answering one question consistently: are two rows observations of the same thing?
Start by deciding what one row represents:
| Record grain | Suitable identity |
|---|---|
| Product family | Source plus merchant product or product-group ID |
| Sellable variant | Product-family identity plus variant ID or source-specific SKU |
| Marketplace offer | Variant identity plus seller and market |
This distinction prevents a new red version of an existing shoe from being reported as an entirely new shoe.
Prefer durable, source-provided identifiers. A practical order is:
- Merchant product or product-group ID.
- Variant ID or source-specific SKU.
- GTIN or manufacturer identifier, stored in its own field.
- A conservatively normalised canonical URL.
- A documented combination of stable fields when no better identifier exists.
An SKU is normally specific to a merchant's catalogue. The same SKU on two unrelated stores does not necessarily identify the same product, so include the source in the key. Do not match products by title or row position because names and catalogue order can change.
Preserve observed and canonical URLs for provenance, but use stronger identifiers when available. Maintain an alias table linking known URLs and identifiers to the same product so a moved page does not create a launch.
The monitoring scope is equally important. Record the competitor, storefront, country or delivery region, language, currency, category and seller where relevant. A product can be known globally but newly introduced in a particular market. Keeping those scopes separate prevents ordinary localisation from becoming a duplicate global-launch alert.
For a deeper treatment of record grain and identifiers, see how to collect product names, prices and SKUs.
Monitor pages that can reveal new products
Checking known product pages can reveal price, content and availability changes. It cannot discover an entirely new product URL.
A competitor product launch tracker must revisit discovery surfaces such as:
- Category and collection pages
- “New arrivals” pages
- Brand pages
- Stable internal search results
- Pagination, load-more controls and infinite scrolling
- Public XML sitemaps
- An authorised catalogue feed or API when one provides the required coverage
Use more than one route when coverage matters. A “new arrivals” page may provide a fast signal, while a broader category crawl reveals quietly added products and reconciles coverage.
When a useful XML sitemap is available, Web Scraper's Sitemap.xml Link selector can follow standard sitemaps, compressed sitemap files and sitemap indexes while filtering irrelevant URL patterns.
Treat XML sitemap metadata as a discovery hint. The optional <lastmod> value describes when a page changed, not when a product launched. A sitemap may also omit products or contain URLs outside the required scope, so validate its coverage against catalogue pages.
Collect a comparable catalogue snapshot
Each accepted run should produce the same record structure and enough evidence to classify changes later.
Fields collected from the source
| Purpose | Suggested fields |
|---|---|
| Scope | Source, category, country, currency and seller |
| Product identity | Merchant product ID, product-group ID, SKU, GTIN and MPN |
| Variant identity | Variant ID, colour, size, material and configuration |
| Product context | Brand, product name and category path |
| Commercial state | Raw and normalised price, currency and availability |
| Provenance | Observed URL and canonical URL when available |
| Source timing | Explicit release or publication date when supplied |
Fields added by the monitoring system
| Purpose | Suggested fields |
|---|---|
| Observation | observed_at and run ID |
| Pipeline audit | Sitemap, extractor and normalisation versions |
| History | first_seen_at, last_seen_at and current status |
| Event handling | Classification, confidence and confirmation time |
Preserve raw values alongside cleaned fields. If a retailer changes “Out of stock” to “Temporarily unavailable”, the raw value shows what the page displayed while the normalised value keeps comparisons consistent.
Store four related record types: runs describe each collection; products hold durable identities; observations preserve what the site showed; and events contain the business classification. Rules can then improve without rewriting the original evidence.
Build listing navigation first, then keep product-card and detail-page selectors inside the same record branch. The guide to scraping listing and detail pages explains how to prevent a name from one product being paired with the SKU of another. If selecting a size or colour changes the identity, price or availability, create variant-level records and test the intended combinations using the guide to clicking product variants.
Establish an accepted baseline
The first complete run establishes what already exists. Validate it, store the identities in history and suppress launch alerts so existing products do not become false launches.
Before accepting the baseline:
- Confirm that the intended categories and pagination paths were covered.
- Test standard, discounted, unavailable and variant products.
- Repeat collection until coverage and identity-field population are reasonably stable.
- Check total records, distinct identities and duplicate rates.
- Confirm the expected region, language and currency.
- Record the sitemap and normalisation versions.
Maintain a persistent historical registry rather than replacing it with yesterday's file. If a product disappears and later returns, history allows the system to classify it as a relist rather than a launch.
When the scope expands into another category or country, seed the first accepted run for that scope as another baseline. A material selector improvement can also reveal old products for the first time. Mark those records as backfilled unless other evidence supports a recent launch.
Compare each healthy run with accepted history
For every current record:
- Normalise identifiers and variant attributes.
- Deduplicate repeated records within the run.
- Resolve the strongest available identity.
- Check exact matches in the historical registry.
- Check known aliases and alternative URLs.
- Check whether the parent product family already exists.
- Compare previous availability and regional scope.
- Create a candidate only when the resulting state changed meaningfully.
- Store the accepted observation and any confirmed aliases.
Three questions handle most cases:
| Question | What it reveals |
|---|---|
| Was the identity absent from the previous snapshot? | It appears newly added in this run |
| Has the identity ever appeared in accepted history? | It is either genuinely unseen or a relist |
| Does its product family already exist? | It is either a new family or a new variant |
For the broader history and comparison architecture, see incremental web scraping and change detection.
Worked example: one launch and three lookalikes
Suppose a footwear retailer monitors a competitor's running-shoe category.
The first accepted run becomes the silent baseline:
| Identity | Baseline state |
|---|---|
shoe-100, blue, size 42 |
In stock |
shoe-200, black, size 41 |
Out of stock |
shoe-300, silver, size 43 |
In stock |
No launch alerts are sent.
The next accepted run contains these observations:
| Observation | Historical evidence | Result |
|---|---|---|
Product family shoe-400 appears |
Neither the family nor its variants appeared before | Genuine product launch |
A red variant appears under shoe-100 |
The family exists, but the variant does not | New variant |
shoe-200, black, size 41 becomes available |
The same variant existed previously | Restock |
shoe-300, silver, size 43 is absent |
It was present, but one absence is inconclusive | Missing observation |
After several accepted runs, shoe-300 returns. Its identity exists in history, so classify it as a relisted product. Repeated accepted absence or a separate source signal would be needed before calling it discontinued.
Only shoe-400 enters the net-new launch feed. The other events may still matter, but they belong in different alert categories.
Validate before applying any change
A completed scraping job is not automatically a correct catalogue snapshot.
A broken selector, incomplete pagination, consent page or access response can create apparent additions and removals. Even 200 OK does not prove that the intended product page or expected fields were returned. The guide to 200 OK but no data explains these failure layers.
Apply acceptance checks before comparison:
- Minimum total record count
- Plausible counts by category and region
- Maximum failed and empty page percentages
- Minimum product-ID, name and URL population
- Expected category and pagination-path coverage
- Plausible distinct-identity and duplicate rates
- Consistent currency, language and regional settings
- Positive markers showing that the expected page type was collected
- Unusual changes relative to recent accepted runs
Web Scraper Cloud's data-quality controls can monitor record counts, failed pages, empty pages and field population. The downstream system should add identity, scope and business-specific checks.
Quarantine a suspicious run instead of letting it replace the last accepted state. Retaining yesterday's trusted catalogue is safer than reporting a mass launch caused by duplicated cards, missed pagination or a redesigned page.
Set a cadence around the business decision
Choose the cadence according to catalogue volatility and the time available to act.
| Need | Suggested approach |
|---|---|
| Same-day visibility in priority categories | Several focused discovery runs per day |
| Ordinary active catalogue | Daily collection |
| Slow-moving specialist catalogue | Several runs per week |
| Coverage and removal assurance | Slower full-catalogue reconciliation |
Detection time includes the wait until the next run, crawl duration, validation and downstream delivery. A daily schedule provides daily checking, not guaranteed immediate detection.
Do not schedule a job more frequently than it can finish. Web Scraper Cloud's scheduler supports recurring schedules, while a new scheduled run waits if the previous one is still active.
Use Fast when the tested sitemap can collect the required fields from returned HTML. Use FullJS when the catalogue needs JavaScript rendering, scrolling or supported interactions. Follow applicable site terms, access restrictions and traffic limits, and coordinate request volume across all jobs. The guide to ethical web-scraping request rates provides operational guidance.
Technical accessibility is not blanket permission to collect or reuse data. Assess relevant terms, robots directives, privacy, copyright, contractual obligations and applicable law for the project. This article provides workflow guidance, not legal advice.
Send evidence instead of raw changes
A useful alert should include:
- Classification and confidence
- Competitor, category and market
- Product-family and variant identifiers
- Product name, availability and observed price
- Product URL
- First-observed and source-supplied dates
- Run and extractor versions
- The rule that produced the classification
Route priority product families immediately, send variants and lower-priority additions in a digest, and hold weak identities or suspicious run-level changes for review. Suppress baselines, known relists and confirmed backfills unless requested.
Where speed matters and the identity is strong, one healthy observation may be enough. When false positives are more costly than a short delay, require the candidate to appear in a second accepted run. Assign each emitted event a repeat-safe key so reprocessing a file does not send the same alert twice.
Begin with one important competitor and a limited category set. Track why candidates were rejected, then refine identifiers, aliases, scope and health thresholds before expanding coverage.
Where Web Scraper fits
Web Scraper provides the collection and delivery layer:
- Build and test the sitemap in the Web Scraper browser extension.
- Verify catalogue navigation, records and variants.
- Move the tested sitemap to Web Scraper Cloud.
- Run it on the required schedule.
- Apply collection-level data-quality controls.
- Export each completed dataset to the downstream environment.
The downstream database, script or analytics workflow maintains historical identities, compares accepted snapshots, classifies launch events and sends business alerts.
That boundary matters because “new product” is not a field that can be extracted from every page. It is a conclusion based on scope, product grain, accepted history and the organisation's own launch rules.
Build the collection layer for reliable launch detection
Create and validate a structured competitor-catalogue sitemap in the free Web Scraper browser extension, then move the tested workflow to Web Scraper Cloud for recurring collection and snapshot delivery.