Can AI shopping agents actually read your product pages?

structured data, product variants, e-commerce data, AI shopping agents, data quality

AI shopping agents can read product pages, but the information they obtain depends on the route they use. Returned HTML, a rendered browser page and a merchant feed can expose different facts about the same product.

For retailers, the useful test is whether that route supplies the right variant, applicable price, stock status and purchase conditions. Seeing your product name in an AI answer establishes very little about the rest of the offer.


How AI shopping agents reach your product pages

Start with the service and the product facts you want to check. “AI-readable” is too broad to be a useful test result.

Retrieval route What to inspect What the check establishes
Direct page fetch Returned HTML and embedded product data Which facts that request received before page JavaScript ran
Interactive browser Rendered page and selected variant or delivery state Which facts the tested browser workflow could reach
Merchant feed or catalogue integration Submitted records, processing results and update times What you supplied and, where reported, what the channel accepted
Search or an existing index Returned product information, cited sources and freshness What that particular lookup surfaced

One service can use several routes. OpenAI has described shopping research reading public retail product pages directly. Its current shopping documentation also describes merchant information supplied through third parties, merchants and Shopify Catalog, alongside a direct-feed option. A shopping answer cannot therefore be assumed to come from a fresh page visit.

Keep three observations separate: what the server returned, what the retrieval tool extracted and what reached the model. A JSON-LD block can exist in a response without appearing in a particular tool's text output. When you control the retrieval pipeline, inspect that output. If the provider does not expose this stage, leave it unverified.

Define the offer before testing its readability

Choose one variant, seller and market as the comparison unit. Record its identifier, selected options, currency, tax basis and observation time. Preserve conditions such as a member-only price, pack quantity or delivery destination beside the value they qualify.

Include the attributes a buyer would use to choose the item. “Capacity: 30 L” supplies a checkable specification; “room for every adventure” leaves rather more to interpretation. Keep important specifications and restrictions available as labelled text rather than relying entirely on product imagery.

Missing information needs its own meaning. A blank stock field means stock was not established. A missing shipping charge does not establish free delivery. OpenAI's product-feed specification makes these distinctions explicit, including an unknown availability value and stable identifiers for individual items or variants. Those are provider-specific fields, but the distinction is useful in any audit.

Audit one product across the routes you support

Use an approved commerce-system record as the expected result, then choose a small sample covering your actual templates and exceptions: a simple product, a promotion, different variants, a sold-out item and location-dependent delivery. This diagnoses the sample. Catalogue-wide coverage requires reconciling results against your product and variant inventory.

1. Confirm that the intended page arrives

Inspect the requested URL, final URL, response status, content type and body. Search for a distinctive SKU and check which product owns it. A recommendation card containing that SKU cannot stand in for the main product.

A successful status can accompany a challenge, consent screen or empty application shell. The guide to 200 OK responses with unusable data explains this failure mode.

Sold-out and discontinued products need an explicit page policy. These synthetic cases use the fictional large bag, BAG-L, whose current offer is $69 USD and sold out.

URL behaviour What a retrieval route receives What to check
The large-bag URL returns 200 with “Sold out”, while markup still says InStock Conflicting availability statements for the same variant Confirm that visible text and the relevant offer markup describe the same stock state.
The URL returns 404 after the bag sells out A missing page instead of an offer reported as unavailable Check whether removing the offer is the intended policy for temporary stock loss or discontinuation.
The URL redirects to the category page A final page that no longer identifies BAG-L Inspect the redirect destination and confirm whether the product identity survives.
The URL returns 200 with an empty product shell A document without enough facts to establish the large-bag offer Compare the initial response with the rendered page and investigate missing data.

Decide on purpose which behaviour each product state should produce, then check that each route receives it.

For a particular shopping service, check its current access guidance and investigate verified provider traffic in your CDN or server logs. OpenAI, for example, documents separate controls for its search crawler and training crawler. Changing a local request's user-agent string tests one request header; it does not reproduce the provider's network, session or processing.

2. Compare the initial response with the selected page

Open browser Developer Tools, select Network and reload the product URL. Select the main document request and inspect Response. Search for the expected identifier, price, currency and stock value, including relevant embedded data. Then inspect the live page in Elements after rendering and after any required variant or destination selection.

Record whether each required field arrived initially, appeared after rendering, required an interaction or remained unresolved. The guide to finding where a website gets its data provides a deeper walkthrough.

Your browser's response establishes what that request received. It does not establish what an external shopping service received.

3. Check structured data and feed identity

Find the Product and Offer information for the intended variant. A block can legitimately describe several variants; check the entry belonging to the target. A product family's lowest price, another seller's offer and a selected variant's price describe different things.

Check page-head signals too: the canonical link, link[rel="canonical"], og:url and similar meta tags can identify a different product state. In a synthetic case, /products/travel-bag?variant=BAG-L declares /products/travel-bag as its canonical, and that parent URL renders the small bag. A retrieval route that follows that signal may record the default variant's $49 price and In stock status. Check whether each variant URL intended to be read independently declares a canonical that serves that variant, or whether the product-group relationship is expressed consistently instead. This does not establish how any particular provider handles canonicals.

To collect these non-visible signals with an Element attribute selector, type the selector manually. Use Attribute name href for link[rel="canonical"] and content for meta[property="og:url"]. Missing attributes return empty values and need an unresolved finding.

Use the channel's own documented requirements. Google's product-variant guidance describes grouping related variants and linking to a preselected variant with the correct image, price and availability. It also recommends initial-HTML Product markup and warns that dynamically generated markup can make Shopping crawls less frequent and less reliable. This is Google-specific guidance, not a universal description of AI agents.

A successful structured-data test leaves factual checks open. Google's guidelines require relevant, up-to-date information and explain that valid markup does not guarantee display. Check offer identity and freshness alongside syntax.

Compare the corresponding feed record by stable identity. Keep its submission or update time separate from the time you collected the page. Inspect processing diagnostics where available; possession of a correct local feed file does not establish that a channel accepted it.

4. Write a finding that identifies the repair

Consider a fictional bag shop. Its small bag costs $49 and is in stock. Its large bag, BAG-L, costs $69 and is sold out. Those are the expected current values in the shop's commerce system.

The following observations are invented to illustrate the audit. They concern a request for the large-variant URL and a feed-file check in the same audit window.

Representation Variant described Price Availability
Initial HTML Small bag $49 USD In stock
Rendered page after URL-based variant selection Large bag, BAG-L $69 USD Sold out
Embedded Product/Offer markup Small bag $49 USD In stock
Latest supplied feed record Large bag, BAG-L $59 USD In stock

The small-bag values are legitimate. They cannot establish the large bag's offer. A collector combining the large SKU with the small price and stock would create an offer the shop cannot fulfil.

Repair each demonstrated fault at the relevant source:

Finding Repair and recheck
Large offer requires rendering If the supported route needs it without JavaScript, expose the intended variant or an explicitly identified set of variants in initial HTML. Check a fresh response.
Markup describes only the small bag Check whether it should also describe the large variant and product-group relationship. Verify that each offer belongs to the correct SKU.
Feed identifies the large bag but conflicts with its price and stock Trace the source mapping and update process, correct the record and check subsequent processing.
Delivery cost needs a postcode that has not been supplied Keep it unresolved until the required destination is tested.
A separate request receives a challenge or wrong regional page Investigate the intended CDN rule, consent handling or redirect, then confirm the expected product response through that route.

Record these as separate findings, then recheck the same URL, state and fields after each repair.

Repeat the page checks with Web Scraper

Once the page structure and required state are understood, Web Scraper can repeat the collection of defined product fields.

Build and test a sitemap in the browser extension. Collect the identifier, selected options and offer values together. Where an interaction is necessary, configure and test an Element click selector. Check whether the data is already accessible before adding a click.

An HTML selector can preserve the contents of an identified structured-data script. Interpret complex JSON relationships, join feed records and calculate differences downstream in your own code or spreadsheet.

Check catalogue coverage from your XML sitemap

A product absent from the XML sitemap is missing from that discovery route before its readability can be checked. A listed URL that redirects away from the product or returns 404 also needs investigation. Other channels may discover the product separately, so keep the finding tied to the route tested.

The Sitemap.xml Link selector reads standard XML sitemaps, compressed .xml.gz files and sitemap indexes. Supply the sitemap through Sitemap URLs or use Discover from robots.txt. Its child selectors run on each listed page.

This synthetic selector tree uses the fictional bag shop's data-product-id marker and product URL pattern. The field names are example configuration, not predefined output columns.

_root
└── sitemap_xml      Sitemap.xml Link: URL regular expression /products/
    ├── sku          Element attribute: [data-product-id], data-product-id
    ├── canonical    Element attribute: link[rel="canonical"], href
    ├── price        Text
    ├── availability Text
    └── product_jsonld  HTML: script[type="application/ld+json"]

The URL regular expression /products/ keeps listed URLs containing that product-path pattern. Use a pattern that actually separates the shop's product pages from other page types; it does not validate their contents.

For sites whose URLs do not separate page types, put a page wrapper under sitemap_xml: an Element selector matching body:has([data-product-id]), with the field selectors beneath it. If the marker is absent, the wrapper does not match and Cloud records the URL as an Empty page.

Children execute inside the matched element. A body wrapper cannot reach link[rel="canonical"] or JSON-LD placed in the page head. Collect head signals in a separate pass and join them to the validated product records downstream. Prefer the URL regular expression where URLs distinguish page types.

Downstream, compare collected SKUs against the commerce system's expected variant list. Missing SKUs are coverage gaps for that audit; Empty pages identify listed URLs that produced no product record. A sitemap containing only a parent product URL may still leave required variants uncollected.

Extract simple offer fields from the captured JSON-LD

For a known block containing one intended offer, create a Virtual column named schema_price from the raw product_jsonld column. Add a Regex match parser with Regex "price"\s*:\s*"?([0-9.]+), select capture Group 1 under Groups and leave Multiple disabled. Enter the expression without surrounding / characters.

Create a second Virtual column, schema_availability, from the same raw source. Use Regex "availability"\s*:\s*"([^"]+)" with Group 1. Virtual columns use raw source-column values, so parsers applied to product_jsonld do not alter these inputs.

Synthetic source snippet from the fictional small bag:

{
  "sku": "BAG-S",
  "offers": {
    "price": "49.00",
    "priceCurrency": "USD",
    "availability": "InStock"
  }
}

Synthetic parser result:

schema_price: 49.00
schema_availability: InStock

This price expression is deliberately limited to the simple numeric format shown. It does not validate arbitrary JSON or interpret offer ownership. With several offers in one block, Regex match with Multiple enabled and a \n Separator can return all matching prices, but it cannot reliably tie each price to its SKU. Pair offers with variants downstream.

Use + Add time scraped with an explicit time zone and format so every exported row carries its observation time.

Move the validated sitemap to Cloud and test the intended configuration there. The Fast and FullJS drivers provide different execution routes: Fast extracts returned HTML without page JavaScript; FullJS executes JavaScript and supports interactions. A sitemap containing scrolling, Element Click or Website State Setup cannot run with Fast. Use a separate compatible sitemap when comparing against an interaction-based workflow.

Measure what depends on rendering

Run the same interaction-free sitemap with Fast and FullJS over the same known URL set. Multiple start URLs can be imported, and each starts from _root. Keep the market, selectors and relevant page state consistent, then compare returned records and required fields.

This synthetic comparison uses 40 product URLs from the fictional bag catalogue, added as multiple start URLs. Each run returns 40 SKU-bearing records.

Field Fast filled / expected Fast fill rate FullJS filled / expected FullJS fill rate
sku 40/40 100% 40/40 100%
price 10/40 25% 40/40 100%
availability 10/40 25% 38/40 95%
product_jsonld 40/40 100% 40/40 100%

The synthetic calculation is filled rows ÷ 40 × 100: 10 ÷ 40 × 100 = 25%, while 38 ÷ 40 × 100 = 95%.

In this synthetic case, inspecting the responses confirms that the additional price and availability values arrive only after JavaScript executes. The two availability fields still unfilled under FullJS need separate investigation. A populated product_jsonld field establishes that a block was captured; its contents still need offer checks.

A Fast run reflects the returned HTML of Web Scraper's own request. It does not reproduce a shopping service's request.

Test each market separately

Location can change currency, prices, availability, the regional catalogue, consent screens and page structure. Run the same sitemap through the required Proxy configuration locations as separate jobs, then add each job's configured location as a column downstream. Datacenter locations are United States, United Kingdom and Germany; Residential proxies provide options for other countries and locations.

Some sites set the market or delivery destination through page state instead of IP location. Website State Setup can establish that state and requires FullJS. Its action sequence runs only when the Perform when not found selector does not match. A condition such as div.store-location:contains('New York') checks for that specific displayed location.

Use Test scrape after each proxy or state-setup change. If a required Website State Setup element is missing, Cloud records a Failed page. A page that extracts successfully in the wrong market can still return rows with the wrong currency, so validate the values as well as row counts.

Synthetic market observations for the fictional large bag:

SKU Configured location Price Currency Availability
BAG-L United States $69 USD Sold out
BAG-L United Kingdom £55 GBP Sold out

These synthetic records describe different market offers. Their different prices are not a mismatch. Compare each with the commerce record and feed for its own market.

Keep URLs, identifiers, selected state and observation times with the results, adding audit context downstream where necessary. Preserve original field values before normalising them. Account for every expected item as matched, mismatched, excluded or unresolved, with a reason. These are proposed audit outcomes, separate from the product's stock status.

Compare the export with commerce and feed records

The synthetic sample below reconciles five exported records with a sixth expected item that produced no row. That missing case is added downstream so the product remains visible in the audit. Columns combine collected fields with URL and market context added downstream.

All entries belong to the fictional small/large bag family. The blue and archived identifiers are synthetic variants. NULL means missing evidence, not a literal string.

sku market url canonical price_raw price currency availability time_scraped
BAG-S US /products/travel-bag?variant=BAG-S /products/travel-bag?variant=BAG-S $49.00 49.00 USD in_stock 2026-11-06T12:00:00Z
BAG-L US /products/travel-bag?variant=BAG-L /products/travel-bag?variant=BAG-L $69.00 69.00 USD sold_out 2026-11-06T12:00:00Z
BAG-S-BLUE US /products/travel-bag?variant=BAG-S-BLUE /products/travel-bag?variant=BAG-S-BLUE $49.00 49.00 USD NULL 2026-11-06T12:00:00Z
BAG-L-BLUE US /products/travel-bag?variant=BAG-L-BLUE /products/travel-bag $69.00 69.00 USD sold_out 2026-11-06T12:00:00Z
BAG-L-OLD US NULL NULL NULL NULL NULL NULL NULL
BAG-L UK /uk/products/travel-bag?variant=BAG-L /uk/products/travel-bag?variant=BAG-L £55.00 55.00 GBP sold_out 2026-11-06T12:05:00Z

In the synthetic missing-row case, Inspect identifies /products/travel-bag?variant=BAG-L-OLD as a 404 Empty page. Its SKU and market above come from the expected inventory; there is no corresponding exported record.

For this synthetic audit, the shop intends each listed variant URL to identify that variant independently. Its expected canonical therefore includes the variant. The parent canonical for BAG-L-BLUE conflicts with that declared contract; a consistently implemented product-group canonical would require a different expectation.

Use these synthetic reference records for commerce_expected:

sku,market,canonical,price,currency,availability
BAG-S,US,/products/travel-bag?variant=BAG-S,49.00,USD,in_stock
BAG-L,US,/products/travel-bag?variant=BAG-L,69.00,USD,sold_out
BAG-S-BLUE,US,/products/travel-bag?variant=BAG-S-BLUE,49.00,USD,in_stock
BAG-L-BLUE,US,/products/travel-bag?variant=BAG-L-BLUE,69.00,USD,sold_out
BAG-L-OLD,US,/products/travel-bag?variant=BAG-L-OLD,69.00,USD,sold_out
BAG-L,UK,/uk/products/travel-bag?variant=BAG-L,55.00,GBP,sold_out

Synthetic feed records for the same audit window:

sku,market,price,currency,availability
BAG-S,US,49.00,USD,in_stock
BAG-L,US,59.00,USD,in_stock
BAG-S-BLUE,US,49.00,USD,in_stock
BAG-L-BLUE,US,69.00,USD,sold_out
BAG-L-OLD,US,69.00,USD,sold_out
BAG-L,UK,55.00,GBP,sold_out

The synthetic explicit_exclusions table contains one deliberate scope decision:

sku,market,reason
BAG-L,UK,Outside this US comparison scope

Run the comparison downstream in a database or script. Convert blank values to SQL NULL, use decimal prices and normalise stock labels consistently. Here, “In stock” becomes in_stock and “Sold out” becomes sold_out. Resolve relative canonical URLs consistently before comparison; the synthetic data shows paths without the common store origin.

Keep one row per sku and market in each compared snapshot. Resolve duplicates before joining. This standard SQL query starts from the expected commerce inventory, so a product missing from the export still receives an outcome:

SELECT
    e.sku,
    e.market,
    CASE
        -- Exclusions are explicit audit decisions, not extraction failures.
        WHEN x.sku IS NOT NULL THEN 'excluded'
        -- An absent row or required value leaves the check unresolved.
        WHEN e.sku IS NULL OR e.market IS NULL
          OR e.canonical IS NULL OR e.price IS NULL
          OR e.currency IS NULL OR e.availability IS NULL
          OR s.sku IS NULL OR s.url IS NULL OR s.canonical IS NULL
          OR s.price_raw IS NULL OR s.price IS NULL
          OR s.currency IS NULL OR s.availability IS NULL
          OR s.time_scraped IS NULL
          OR f.sku IS NULL OR f.price IS NULL
          OR f.currency IS NULL OR f.availability IS NULL
          OR e.availability = 'unknown'
          OR s.availability = 'unknown'
          OR f.availability = 'unknown' THEN 'unresolved'
        -- Compare prices only after all three currencies agree.
        WHEN e.currency <> s.currency OR e.currency <> f.currency
            THEN 'unresolved'
        WHEN e.canonical <> s.canonical
          OR e.price <> s.price OR e.price <> f.price
          OR e.availability <> s.availability
          OR e.availability <> f.availability THEN 'mismatched'
        ELSE 'matched'
    END AS audit_outcome
FROM commerce_expected AS e
LEFT JOIN scraped_export AS s
    ON s.sku = e.sku AND s.market = e.market
LEFT JOIN feed AS f
    ON f.sku = e.sku AND f.market = e.market
LEFT JOIN explicit_exclusions AS x
    ON x.sku = e.sku AND x.market = e.market
ORDER BY e.market, e.sku;

Explicit exclusions take precedence. For included items, missing required evidence or unequal currencies produces unresolved before any price comparison. Price and availability agree only when the commerce record, export and feed agree.

Synthetic results:

sku market audit_outcome Reason
BAG-S US matched Page, expected offer, canonical and feed agree.
BAG-L US mismatched Page and commerce record show $69 and sold out; feed shows $59 and in stock.
BAG-S-BLUE US unresolved Exported availability is blank.
BAG-L-BLUE US mismatched Canonical points to the parent instead of the variant required by this audit.
BAG-L-OLD US unresolved Expected product has no exported row; Inspect reports the 404.
BAG-L UK excluded Explicitly outside this US comparison scope.

The UK row is excluded because of the stated scope decision, not because £55 differs from $69. Compare it against UK commerce and feed records in a UK audit.

After validating the Cloud output, schedule the tested collection and compare exported results downstream. Repeat checks after template releases, feed-mapping changes and promotions. A completed scraping job still requires checks of required fields and coverage.

Turn regressions into alerts

Configure Cloud's Data quality control checks around the validated collection. Fill with suggested values uses a previous job to provide a starting baseline; review those values against the catalogue and states the next run should cover.

Symptom Possible cause Next check
Maximum failed page % rises 403, 429, CAPTCHA, timeout or a required state-setup element is missing Use Inspect to review the URL, failure reason, preview and downloadable page HTML. Access challenges also suggest that other automated clients may be challenged; their access needs separate verification.
Maximum empty page % rises Sold-out URLs now return 404, or the page-wrapper marker was removed Compare the Empty URLs with the expected variant inventory and inspect the product marker.
Minimum % of fields that must be filled drops for price after a template release The selector stopped matching, or price now arrives through JavaScript Inspect the required field and compare Fast and FullJS output under the same market conditions.
Minimum record count drops The XML sitemap shrank, or required variants were removed Compare sitemap coverage and collected SKUs with the expected commerce inventory.
No value pages appear Elements matched, but no populated values reached the output Inspect the received HTML and extraction rules. A single blank field in an otherwise populated record still needs its own downstream check.

Cloud automatically retries Empty and Failed pages. Inspect the remaining outcomes; Fast driver jobs do not produce screenshots.

A Data quality failure does not turn a completed job into a failed job. Configure Email or Cloud notifications to alert on the quality failure, then investigate the affected records.

Combine Scheduler with automatic Data Export to Google Sheets or Amazon S3 after each job. Those fresh exports can supply a downstream spreadsheet, database or script comparison; the joining and audit classification remain downstream.

Verify the shopping channel's result separately

After repairing the sources you control, run a lookup in the intended service. Preserve the prompt, cited sources, displayed variant, decisive values and time. Where available, compare merchant diagnostics and verified access records with that observation.

Treat it as evidence of that lookup. An absent product alone cannot identify a crawling failure, and one accurate result cannot establish catalogue coverage. OpenAI's shopping documentation also warns that pricing and shipping updates can take time to appear. Separate a delayed downstream result from a mismatch that still exists on your page or in your feed.

Web Scraper gives you repeatable evidence about what your configured workflow collected. The shopping service's model input and recommendations require their own checks. Begin with one product family, fix the first demonstrated fault and extend the audit across the variants and markets you need to support.

Explore e-commerce data collection with Web Scraper Cloud to turn those recurring page checks into structured datasets.


Go back to blog page