Scrape listing pages

Target structure

A listing page contains repeated wrapper elements. Each wrapper contains fields such as name, price, URL, image, location or status.

Create the sitemap

  1. Use the listing page as the start URL.
  2. Create an Element selector that matches every item wrapper.
  3. Add child selectors for each field inside the wrapper.
  4. Use Text selectors for visible text, Image selectors for image URLs, Link selectors for URLs and Element attribute selectors for data attributes.

Example selector tree

_root
└── item
    ├── name
    ├── price
    ├── item_url
    └── image_url

Validation

  • Make sure the Element selector matches the repeated records you intend to scrape.
  • Avoid position-dependent selectors when possible, such as :nth-child() or :nth-of-type(), as changes in the page structure can cause them to match the wrong element.
  • Check whether advertisements, recommendation cards, or other repeated blocks are also matched, and adjust the selector to include or exclude them as needed.

Add additional pages

Use a Pagination selector to navigate numbered pagination, Next or arrow controls, and Load more buttons. For standard pagination that provides both numbered controls and a Next or arrow control, target the Next or arrow control where possible because it is generally the more stable option. For infinite-scroll listings, enable scrolling on the Element selector.

If a listing uses both pagination and scrolling to render records, place the scrolling Element selector beneath the Pagination selector. The Pagination selector traverses the pages recursively, and the child Element selector performs its scrolling behavior on each page before extracting the records.

Related