Start URLs and URL ranges

Choose the start URL

Use the page where the first navigation or extraction selector should run. This is commonly a category page, search result, directory page or individual detail page.

Use multiple start URLs

Add multiple start URLs when the same selectors should be used on several pages. Each start URL starts from _root.

Avoid adding thousands of URLs manually when the website already provides navigation, a predictable URL pattern, or an XML sitemap.

Fig. 1: Import start URLs

Use a numeric URL range

Replace the changing numeric segment with a range definition.

https://example.com/page/[1-3]

This generates:

  • https://example.com/page/1
  • https://example.com/page/2
  • https://example.com/page/3

Use zero padding

https://example.com/archive/[001-100]

Use zero padding only when the source URL contains the padded number.

Use an increment

https://example.com/results?offset=[0-100:10]

This produces offsets of 0, 10, 20 and so on.

Validation

  • Open the first, middle and last generated URL manually.
  • Confirm that invalid ranges do not return generic error or redirect pages.
  • Confirm that every start page uses the same selector structure.

Related

Related videos