Scrape listing and detail pages

Target structure

The listing page contains repeated items and a link to a detail page. Some fields are available in the listing, while others exist only on the detail page.

Fig. 2: News site

Create the listing record

  1. Create an Element selector for the repeated listing items.
  2. Add listing-level fields beneath the Element selector.
  3. Add a Link selector for the detail-page link beneath the same wrapper.

Add detail-page fields

  1. Open the Link selector in the selector tree.
  2. Add the selectors that should run on the destination page.
  3. Preview the fields on several detail pages.

Example selector tree

_root
└── item
    ├── listing_name
    ├── listing_price
    └── detail_link
        ├── sku
        ├── description
        └── availability

Fig. 3: News site sitemap

Common errors

  • Detail selectors are attached to the listing wrapper instead of the Link selector.
  • The Link selector extracts the URL but is not configured as the parent of detail fields.
  • The item wrapper includes more than one detail link.
  • Detail pages use multiple layouts and require a broader or conditional selector.

Related