Scrape URLs from sitemap.xml

When to use this workflow

Use a Sitemap.xml Link selector when the website publishes the target URLs in standard XML sitemaps and page navigation would otherwise be complex or incomplete.

Configure the selector

  1. Use the website homepage as the sitemap start URL.
  2. Create a Sitemap.xml Link selector.
  3. Add the XML sitemap URL or discover sitemap URLs from robots.txt.
  4. Add a URL regular expression to include only the required page type. (optional)
  5. Add the detail-page extraction selectors as children.

Filter page types

XML sitemaps often contain categories, products, articles and utility pages. Filter by a stable URL segment such as /product/ when only one page type should be scraped.

Use a page wrapper when URLs are inconsistent

A page wrapper is an advanced manual-selector pattern for cases where an XML sitemap contains several page types but the URLs do not have a reliable pattern that can be used to distinguish them. For example, the same XML sitemap may contain listing pages, blog pages and product pages with no consistent URL structure separating those page types. In this case, use an Element selector as a structural page-type check and place the extraction selectors beneath it.

Example

Assume an XML sitemap contains listing pages, blog pages and product pages, but the URL structure cannot reliably distinguish between them. Product pages always contain h1.product-title.

_root
└── sitemap_xml
    └── product_page    Element: body:has(h1.product-title)
        ├── title       Text: h1.product-title
        ├── price       Text: .price
        └── sku         Text: .sku

The product_page Element selector acts as a conditional wrapper. On a product page, body:has(h1.product-title) matches the page body and the child selectors execute inside that matched element. On a listing page, blog page or any other page where h1.product-title does not exist, the wrapper does not match and its child selectors do not execute. In Web Scraper Cloud, the visited URL is therefore returned as an Empty page because no record was extracted from that page.

Why :has() is useful

The :has() condition lets the wrapper match an element based on content that exists somewhere inside it. In this example, the selector does not extract the title itself. It asks whether the page body contains the product-title element. The body is returned only when that condition is true.

This makes the wrapper particularly useful when the page type can be identified from its HTML structure but not from its URL.

Choose a reliable page marker

The element used inside :has() should identify the page type reliably. Prefer a marker that exists on every valid target page and does not appear on unrelated pages.

  • body:has(h1.product-title) can work when every product page has that heading and non-product pages do not.
  • body:has([data-product-id]) can be stronger when the site exposes a product-specific attribute.
  • A generic element such as body:has(.price) may be too broad if prices also appear in recommendations, category listings or advertisements.

Do not use an optional field as the page marker. If some valid product pages do not contain the selected marker, those pages will be excluded completely because the wrapper will not match.

Create the wrapper manually

  1. Navigate into the selector that opens the mixed set of target URLs, such as a Sitemap.xml Link selector or Link selector.
  2. Create an Element selector manually.
  3. Enter a selector that matches the page only when the required page marker exists, for example body:has(h1.product-title).
  4. Use Element Preview to confirm that the wrapper matches on a valid target page.
  5. Test a non-target page and confirm that the wrapper does not match.
  6. Navigate into the wrapper Element selector and create the Text, Image, Element Attribute or other field selectors beneath it.

Why the child-selector structure matters

The filtering behavior comes from the selector tree, not from the field selectors themselves. The extraction selectors must be children of the wrapper. If title, price or other fields remain as siblings of product_page, they can still execute on URLs where the wrapper does not match.

Recommended

_root
└── sitemap_xml
    └── product_page
        ├── title
        └── price

Not equivalent

_root
└── sitemap_xml
    ├── product_page
    ├── title
    └── price

When to use a page wrapper

  • An XML sitemap contains several page types, such as listing, blog and product pages, and URL filtering cannot distinguish them reliably.
  • The target page type has a stable structural marker in its HTML.
  • You want extraction selectors to execute only on pages that match that structural marker.
  • You accept that non-matching URLs in a Cloud job will be returned as Empty pages.

When not to use it

Use a simpler URL filter when the target pages can be identified reliably from their URLs. A page wrapper adds another level to the selector tree and should be used when structural validation is genuinely required.

Also verify that the marker exists when Web Scraper evaluates the page. If the identifying element is added only after an interaction or later page rendering, the wrapper may not match in the initial state.

Large XML sitemaps

Compressed and nested XML sitemaps are supported, but large combined downloads can exceed limits. Split the workflow into multiple sitemaps when necessary.

Related