Can AI shopping agents actually read your product pages?
November 04, 2026
structured data, product variants, e-commerce data, AI shopping agents, data quality
AI shopping agents can read product pages, but the information they obtain depends on the route they use. Returned HTML, a rendered browser page and a merchant feed can expose different facts about the same product.
For retailers, the useful test is whether that route supplies the right variant, applicable price, stock status and purchase conditions. Seeing your product name in an AI answer establishes very little about the rest of the offer.
How AI shopping agents reach your product pages
Start with the service and the product facts you want to check. “AI-readable” is too broad to be a useful test result.
| Retrieval route | What to inspect | What the check establishes |
|---|---|---|
| Direct page fetch | Returned HTML and embedded product data | Which facts that request received before page JavaScript ran |
| Interactive browser | Rendered page and selected variant or delivery state | Which facts the tested browser workflow could reach |
| Merchant feed or catalogue integration | Submitted records, processing results and update times | What you supplied and, where reported, what the channel accepted |
| Search or an existing index | Returned product information, cited sources and freshness | What that particular lookup surfaced |
One service can use several routes. OpenAI has described shopping research reading public retail product pages directly. Its current shopping documentation also describes merchant information supplied through third parties, merchants and Shopify Catalog, alongside a direct-feed option. A shopping answer cannot therefore be assumed to come from a fresh page visit.
Keep three observations separate: what the server returned, what the retrieval tool extracted and what reached the model. A JSON-LD block can exist in a response without appearing in a particular tool's text output. When you control the retrieval pipeline, inspect that output. If the provider does not expose this stage, leave it unverified.
Define the offer before testing its readability
Choose one variant, seller and market as the comparison unit. Record its identifier, selected options, currency, tax basis and observation time. Preserve conditions such as a member-only price, pack quantity or delivery destination beside the value they qualify.
Include the attributes a buyer would use to choose the item. “Capacity: 30 L” supplies a checkable specification; “room for every adventure” leaves rather more to interpretation. Keep important specifications and restrictions available as labelled text rather than relying entirely on product imagery.
Missing information needs its own meaning. A blank stock field means stock was not established. A missing shipping charge does not establish free delivery. OpenAI's product-feed specification makes these distinctions explicit, including an unknown availability value and stable identifiers for individual items or variants. Those are provider-specific fields, but the distinction is useful in any audit.
Audit one product across the routes you support
Use an approved commerce-system record as the expected result, then choose a small sample covering your actual templates and exceptions: a simple product, a promotion, different variants, a sold-out item and location-dependent delivery. This diagnoses the sample. Catalogue-wide coverage requires reconciling results against your product and variant inventory.
1. Confirm that the intended page arrives
Inspect the requested URL, final URL, response status, content type and body. Search for a distinctive SKU and check which product owns it. A recommendation card containing that SKU cannot stand in for the main product.
A successful status can accompany a challenge, consent screen or empty application shell. The guide to 200 OK responses with unusable data explains this failure mode.
Sold-out and discontinued products need an explicit page policy. These synthetic cases use the fictional large bag, BAG-L, whose current offer is $69 USD and sold out.
| URL behaviour | What a retrieval route receives | What to check |
|---|---|---|
The large-bag URL returns 200 with “Sold out”, while markup still says InStock |
Conflicting availability statements for the same variant | Confirm that visible text and the relevant offer markup describe the same stock state. |
| The URL returns 404 after the bag sells out | A missing page instead of an offer reported as unavailable | Check whether removing the offer is the intended policy for temporary stock loss or discontinuation. |
| The URL redirects to the category page | A final page that no longer identifies BAG-L |
Inspect the redirect destination and confirm whether the product identity survives. |
| The URL returns 200 with an empty product shell | A document without enough facts to establish the large-bag offer | Compare the initial response with the rendered page and investigate missing data. |
Decide on purpose which behaviour each product state should produce, then check that each route receives it.
For a particular shopping service, check its current access guidance and investigate verified provider traffic in your CDN or server logs. OpenAI, for example, documents separate controls for its search crawler and training crawler. Changing a local request's user-agent string tests one request header; it does not reproduce the provider's network, session or processing.
2. Compare the initial response with the selected page
Open browser Developer Tools, select Network and reload the product URL. Select the main document request and inspect Response. Search for the expected identifier, price, currency and stock value, including relevant embedded data. Then inspect the live page in Elements after rendering and after any required variant or destination selection.
Record whether each required field arrived initially, appeared after rendering, required an interaction or remained unresolved. The guide to finding where a website gets its data provides a deeper walkthrough.
Your browser's response establishes what that request received. It does not establish what an external shopping service received.
3. Check structured data and feed identity
Find the Product and Offer information for the intended variant. A block can legitimately describe several variants; check the entry belonging to the target. A product family's lowest price, another seller's offer and a selected variant's price describe different things.
Check page-head signals too: the canonical link, link[rel="canonical"], og:url and similar meta tags can identify a different product state. In a synthetic case, /products/travel-bag?variant=BAG-L declares /products/travel-bag as its canonical, and that parent URL renders the small bag. A retrieval route that follows that signal may record the default variant's $49 price and In stock status. Check whether each variant URL intended to be read independently declares a canonical that serves that variant, or whether the product-group relationship is expressed consistently instead. This does not establish how any particular provider handles canonicals.
To collect these non-visible signals with an Element attribute selector, type the selector manually. Use Attribute name href for link[rel="canonical"] and content for meta[property="og:url"]. Missing attributes return empty values and need an unresolved finding.
Use the channel's own documented requirements. Google's product-variant guidance describes grouping related variants and linking to a preselected variant with the correct image, price and availability. It also recommends initial-HTML Product markup and warns that dynamically generated markup can make Shopping crawls less frequent and less reliable. This is Google-specific guidance, not a universal description of AI agents.
A successful structured-data test leaves factual checks open. Google's guidelines require relevant, up-to-date information and explain that valid markup does not guarantee display. Check offer identity and freshness alongside syntax.
Compare the corresponding feed record by stable identity. Keep its submission or update time separate from the time you collected the page. Inspect processing diagnostics where available; possession of a correct local feed file does not establish that a channel accepted it.
4. Write a finding that identifies the repair
Consider a fictional bag shop. Its small bag costs $49 and is in stock. Its large bag, BAG-L, costs $69 and is sold out. Those are the expected current values in the shop's commerce system.
The following observations are invented to illustrate the audit. They concern a request for the large-variant URL and a feed-file check in the same audit window.
| Representation | Variant described | Price | Availability |
|---|---|---|---|
| Initial HTML | Small bag | $49 USD | In stock |
| Rendered page after URL-based variant selection | Large bag, BAG-L |
$69 USD | Sold out |
| Embedded Product/Offer markup | Small bag | $49 USD | In stock |
| Latest supplied feed record | Large bag, BAG-L |
$59 USD | In stock |
The small-bag values are legitimate. They cannot establish the large bag's offer. A collector combining the large SKU with the small price and stock would create an offer the shop cannot fulfil.
Repair each demonstrated fault at the relevant source:
| Finding | Repair and recheck |
|---|---|
| Large offer requires rendering | If the supported route needs it without JavaScript, expose the intended variant or an explicitly identified set of variants in initial HTML. Check a fresh response. |
| Markup describes only the small bag | Check whether it should also describe the large variant and product-group relationship. Verify that each offer belongs to the correct SKU. |
| Feed identifies the large bag but conflicts with its price and stock | Trace the source mapping and update process, correct the record and check subsequent processing. |
| Delivery cost needs a postcode that has not been supplied | Keep it unresolved until the required destination is tested. |
| A separate request receives a challenge or wrong regional page | Investigate the intended CDN rule, consent handling or redirect, then confirm the expected product response through that route. |
Record these as separate findings, then recheck the same URL, state and fields after each repair.
Repeat the page checks with Web Scraper
Once the page structure and required state are understood, Web Scraper can repeat the collection of defined product fields.
Build and test a sitemap in the browser extension. Collect the identifier, selected options and offer values together. Where an interaction is necessary, configure and test an Element click selector. Check whether the data is already accessible before adding a click.
An HTML selector can preserve the contents of an identified structured-data script. Interpret complex JSON relationships, join feed records and calculate differences downstream in your own code or spreadsheet.
Check catalogue coverage from your XML sitemap
A product absent from the XML sitemap is missing from that discovery route before its readability can be checked. A listed URL that redirects away from the product or returns 404 also needs investigation. Other channels may discover the product separately, so keep the finding tied to the route tested.
The Sitemap.xml Link selector reads standard XML sitemaps, compressed .xml.gz files and sitemap indexes. Supply the sitemap through Sitemap URLs or use Discover from robots.txt. Its child selectors run on each listed page.
This synthetic selector tree uses the fictional bag shop's data-product-id marker and product URL pattern. The field names are example configuration, not predefined output columns.
_root
└── sitemap_xml Sitemap.xml Link: URL regular expression /products/
├── sku Element attribute: [data-product-id], data-product-id
├── canonical Element attribute: link[rel="canonical"], href
├── price Text
├── availability Text
└── product_jsonld HTML: script[type="application/ld+json"]
The URL regular expression /products/ keeps listed URLs containing that product-path pattern. Use a pattern that actually separates the shop's product pages from other page types; it does not validate their contents.
For sites whose URLs do not separate page types, put a page wrapper under sitemap_xml: an Element selector matching body:has([data-product-id]), with the field selectors beneath it. If the marker is absent, the wrapper does not match and Cloud records the URL as an Empty page.
Children execute inside the matched element. A body wrapper cannot reach link[rel="canonical"] or JSON-LD placed in the page head. Collect head signals in a separate pass and join them to the validated product records downstream. Prefer the URL regular expression where URLs distinguish page types.
Downstream, compare collected SKUs against the commerce system's expected variant list. Missing SKUs are coverage gaps for that audit; Empty pages identify listed URLs that produced no product record. A sitemap containing only a parent product URL may still leave required variants uncollected.
Extract simple offer fields from the captured JSON-LD
For a known block containing one intended offer, create a Virtual column named schema_price from the raw product_jsonld column. Add a Regex match parser with Regex "price"\s*:\s*"?([0-9.]+), select capture Group 1 under Groups and leave Multiple disabled. Enter the expression without surrounding / characters.
Create a second Virtual column, schema_availability, from the same raw source. Use Regex "availability"\s*:\s*"([^"]+)" with Group 1. Virtual columns use raw source-column values, so parsers applied to product_jsonld do not alter these inputs.
Synthetic source snippet from the fictional small bag:
{
"sku": "BAG-S",
"offers": {
"price": "49.00",
"priceCurrency": "USD",
"availability": "InStock"
}
}
Synthetic parser result:
schema_price: 49.00
schema_availability: InStock
This price expression is deliberately limited to the simple numeric format shown. It does not validate arbitrary JSON or interpret offer ownership. With several offers in one block, Regex match with Multiple enabled and a \n Separator can return all matching prices, but it cannot reliably tie each price to its SKU. Pair offers with variants downstream.
Use + Add time scraped with an explicit time zone and format so every exported row carries its observation time.
Move the validated sitemap to Cloud and test the intended configuration there. The Fast and FullJS drivers provide different execution routes: Fast extracts returned HTML without page JavaScript; FullJS executes JavaScript and supports interactions. A sitemap containing scrolling, Element Click or Website State Setup cannot run with Fast. Use a separate compatible sitemap when comparing against an interaction-based workflow.
Measure what depends on rendering
Run the same interaction-free sitemap with Fast and FullJS over the same known URL set. Multiple start URLs can be imported, and each starts from _root. Keep the market, selectors and relevant page state consistent, then compare returned records and required fields.
This synthetic comparison uses 40 product URLs from the fictional bag catalogue, added as multiple start URLs. Each run returns 40 SKU-bearing records.
| Field | Fast filled / expected | Fast fill rate | FullJS filled / expected | FullJS fill rate |
|---|---|---|---|---|
sku |
40/40 | 100% | 40/40 | 100% |
price |
10/40 | 25% | 40/40 | 100% |
availability |
10/40 | 25% | 38/40 | 95% |
product_jsonld |
40/40 | 100% | 40/40 | 100% |
The synthetic calculation is filled rows ÷ 40 × 100: 10 ÷ 40 × 100 = 25%, while 38 ÷ 40 × 100 = 95%.
In this synthetic case, inspecting the responses confirms that the additional price and availability values arrive only after JavaScript executes. The two availability fields still unfilled under FullJS need separate investigation. A populated product_jsonld field establishes that a block was captured; its contents still need offer checks.
A Fast run reflects the returned HTML of Web Scraper's own request. It does not reproduce a shopping service's request.
Test each market separately
Location can change currency, prices, availability, the regional catalogue, consent screens and page structure. Run the same sitemap through the required Proxy configuration locations as separate jobs, then add each job's configured location as a column downstream. Datacenter locations are United States, United Kingdom and Germany; Residential proxies provide options for other countries and locations.
Some sites set the market or delivery destination through page state instead of IP location. Website State Setup can establish that state and requires FullJS. Its action sequence runs only when the Perform when not found selector does not match. A condition such as div.store-location:contains('New York') checks for that specific displayed location.
Use Test scrape after each proxy or state-setup change. If a required Website State Setup element is missing, Cloud records a Failed page. A page that extracts successfully in the wrong market can still return rows with the wrong currency, so validate the values as well as row counts.
Synthetic market observations for the fictional large bag:
| SKU | Configured location | Price | Currency | Availability |
|---|---|---|---|---|
BAG-L |
United States | $69 | USD | Sold out |
BAG-L |
United Kingdom | £55 | GBP | Sold out |
These synthetic records describe different market offers. Their different prices are not a mismatch. Compare each with the commerce record and feed for its own market.
Keep URLs, identifiers, selected state and observation times with the results, adding audit context downstream where necessary. Preserve original field values before normalising them. Account for every expected item as matched, mismatched, excluded or unresolved, with a reason. These are proposed audit outcomes, separate from the product's stock status.
Compare the export with commerce and feed records
The synthetic sample below reconciles five exported records with a sixth expected item that produced no row. That missing case is added downstream so the product remains visible in the audit. Columns combine collected fields with URL and market context added downstream.
All entries belong to the fictional small/large bag family. The blue and archived identifiers are synthetic variants. NULL means missing evidence, not a literal string.
| sku | market | url | canonical | price_raw | price | currency | availability | time_scraped |
|---|---|---|---|---|---|---|---|---|
| BAG-S | US | /products/travel-bag?variant=BAG-S | /products/travel-bag?variant=BAG-S | $49.00 | 49.00 | USD | in_stock | 2026-11-06T12:00:00Z |
| BAG-L | US | /products/travel-bag?variant=BAG-L | /products/travel-bag?variant=BAG-L | $69.00 | 69.00 | USD | sold_out | 2026-11-06T12:00:00Z |
| BAG-S-BLUE | US | /products/travel-bag?variant=BAG-S-BLUE | /products/travel-bag?variant=BAG-S-BLUE | $49.00 | 49.00 | USD | NULL | 2026-11-06T12:00:00Z |
| BAG-L-BLUE | US | /products/travel-bag?variant=BAG-L-BLUE | /products/travel-bag | $69.00 | 69.00 | USD | sold_out | 2026-11-06T12:00:00Z |
| BAG-L-OLD | US | NULL | NULL | NULL | NULL | NULL | NULL | NULL |
| BAG-L | UK | /uk/products/travel-bag?variant=BAG-L | /uk/products/travel-bag?variant=BAG-L | £55.00 | 55.00 | GBP | sold_out | 2026-11-06T12:05:00Z |
In the synthetic missing-row case, Inspect identifies /products/travel-bag?variant=BAG-L-OLD as a 404 Empty page. Its SKU and market above come from the expected inventory; there is no corresponding exported record.
For this synthetic audit, the shop intends each listed variant URL to identify that variant independently. Its expected canonical therefore includes the variant. The parent canonical for BAG-L-BLUE conflicts with that declared contract; a consistently implemented product-group canonical would require a different expectation.
Use these synthetic reference records for commerce_expected:
sku,market,canonical,price,currency,availability
BAG-S,US,/products/travel-bag?variant=BAG-S,49.00,USD,in_stock
BAG-L,US,/products/travel-bag?variant=BAG-L,69.00,USD,sold_out
BAG-S-BLUE,US,/products/travel-bag?variant=BAG-S-BLUE,49.00,USD,in_stock
BAG-L-BLUE,US,/products/travel-bag?variant=BAG-L-BLUE,69.00,USD,sold_out
BAG-L-OLD,US,/products/travel-bag?variant=BAG-L-OLD,69.00,USD,sold_out
BAG-L,UK,/uk/products/travel-bag?variant=BAG-L,55.00,GBP,sold_out
Synthetic feed records for the same audit window:
sku,market,price,currency,availability
BAG-S,US,49.00,USD,in_stock
BAG-L,US,59.00,USD,in_stock
BAG-S-BLUE,US,49.00,USD,in_stock
BAG-L-BLUE,US,69.00,USD,sold_out
BAG-L-OLD,US,69.00,USD,sold_out
BAG-L,UK,55.00,GBP,sold_out
The synthetic explicit_exclusions table contains one deliberate scope decision:
sku,market,reason
BAG-L,UK,Outside this US comparison scope
Run the comparison downstream in a database or script. Convert blank values to SQL NULL, use decimal prices and normalise stock labels consistently. Here, “In stock” becomes in_stock and “Sold out” becomes sold_out. Resolve relative canonical URLs consistently before comparison; the synthetic data shows paths without the common store origin.
Keep one row per sku and market in each compared snapshot. Resolve duplicates before joining. This standard SQL query starts from the expected commerce inventory, so a product missing from the export still receives an outcome:
SELECT
e.sku,
e.market,
CASE
-- Exclusions are explicit audit decisions, not extraction failures.
WHEN x.sku IS NOT NULL THEN 'excluded'
-- An absent row or required value leaves the check unresolved.
WHEN e.sku IS NULL OR e.market IS NULL
OR e.canonical IS NULL OR e.price IS NULL
OR e.currency IS NULL OR e.availability IS NULL
OR s.sku IS NULL OR s.url IS NULL OR s.canonical IS NULL
OR s.price_raw IS NULL OR s.price IS NULL
OR s.currency IS NULL OR s.availability IS NULL
OR s.time_scraped IS NULL
OR f.sku IS NULL OR f.price IS NULL
OR f.currency IS NULL OR f.availability IS NULL
OR e.availability = 'unknown'
OR s.availability = 'unknown'
OR f.availability = 'unknown' THEN 'unresolved'
-- Compare prices only after all three currencies agree.
WHEN e.currency <> s.currency OR e.currency <> f.currency
THEN 'unresolved'
WHEN e.canonical <> s.canonical
OR e.price <> s.price OR e.price <> f.price
OR e.availability <> s.availability
OR e.availability <> f.availability THEN 'mismatched'
ELSE 'matched'
END AS audit_outcome
FROM commerce_expected AS e
LEFT JOIN scraped_export AS s
ON s.sku = e.sku AND s.market = e.market
LEFT JOIN feed AS f
ON f.sku = e.sku AND f.market = e.market
LEFT JOIN explicit_exclusions AS x
ON x.sku = e.sku AND x.market = e.market
ORDER BY e.market, e.sku;
Explicit exclusions take precedence. For included items, missing required evidence or unequal currencies produces unresolved before any price comparison. Price and availability agree only when the commerce record, export and feed agree.
Synthetic results:
| sku | market | audit_outcome | Reason |
|---|---|---|---|
| BAG-S | US | matched | Page, expected offer, canonical and feed agree. |
| BAG-L | US | mismatched | Page and commerce record show $69 and sold out; feed shows $59 and in stock. |
| BAG-S-BLUE | US | unresolved | Exported availability is blank. |
| BAG-L-BLUE | US | mismatched | Canonical points to the parent instead of the variant required by this audit. |
| BAG-L-OLD | US | unresolved | Expected product has no exported row; Inspect reports the 404. |
| BAG-L | UK | excluded | Explicitly outside this US comparison scope. |
The UK row is excluded because of the stated scope decision, not because £55 differs from $69. Compare it against UK commerce and feed records in a UK audit.
After validating the Cloud output, schedule the tested collection and compare exported results downstream. Repeat checks after template releases, feed-mapping changes and promotions. A completed scraping job still requires checks of required fields and coverage.
Turn regressions into alerts
Configure Cloud's Data quality control checks around the validated collection. Fill with suggested values uses a previous job to provide a starting baseline; review those values against the catalogue and states the next run should cover.
| Symptom | Possible cause | Next check |
|---|---|---|
| Maximum failed page % rises | 403, 429, CAPTCHA, timeout or a required state-setup element is missing | Use Inspect to review the URL, failure reason, preview and downloadable page HTML. Access challenges also suggest that other automated clients may be challenged; their access needs separate verification. |
| Maximum empty page % rises | Sold-out URLs now return 404, or the page-wrapper marker was removed | Compare the Empty URLs with the expected variant inventory and inspect the product marker. |
| Minimum % of fields that must be filled drops for price after a template release | The selector stopped matching, or price now arrives through JavaScript | Inspect the required field and compare Fast and FullJS output under the same market conditions. |
| Minimum record count drops | The XML sitemap shrank, or required variants were removed | Compare sitemap coverage and collected SKUs with the expected commerce inventory. |
| No value pages appear | Elements matched, but no populated values reached the output | Inspect the received HTML and extraction rules. A single blank field in an otherwise populated record still needs its own downstream check. |
Cloud automatically retries Empty and Failed pages. Inspect the remaining outcomes; Fast driver jobs do not produce screenshots.
A Data quality failure does not turn a completed job into a failed job. Configure Email or Cloud notifications to alert on the quality failure, then investigate the affected records.
Combine Scheduler with automatic Data Export to Google Sheets or Amazon S3 after each job. Those fresh exports can supply a downstream spreadsheet, database or script comparison; the joining and audit classification remain downstream.
Verify the shopping channel's result separately
After repairing the sources you control, run a lookup in the intended service. Preserve the prompt, cited sources, displayed variant, decisive values and time. Where available, compare merchant diagnostics and verified access records with that observation.
Treat it as evidence of that lookup. An absent product alone cannot identify a crawling failure, and one accurate result cannot establish catalogue coverage. OpenAI's shopping documentation also warns that pricing and shipping updates can take time to appear. Separate a delayed downstream result from a mismatch that still exists on your page or in your feed.
Web Scraper gives you repeatable evidence about what your configured workflow collected. The shopping service's model input and recommendations require their own checks. Begin with one product family, fix the first demonstrated fault and extend the audit across the variants and markets you need to support.
Explore e-commerce data collection with Web Scraper Cloud to turn those recurring page checks into structured datasets.