HTML, CSS and jQuery for Web Scraper

How Web Scraper reads a web page

Web pages are represented as a tree of HTML elements called the Document Object Model, or DOM. Web Scraper selectors locate elements in this tree and then extract data, follow links, group repeated records or interact with the page.

When working in the browser extension, inspect the rendered page in Developer Tools. A website can modify the DOM after the initial HTML loads, so the elements visible in Developer Tools may differ from the original page source.

HTML elements

An HTML element usually consists of a tag, optional attributes and content.

<a class="product-link" href="/products/123" data-sku="ABC-123">
  Example product
</a>

In this example:

  • a is the element type.
  • class="product-link" assigns the element to a class.
  • href="/products/123" contains the link destination.
  • data-sku="ABC-123" is a custom data attribute.
  • Example product is the element's text content.

Common HTML elements

Element Typical use Web scraping relevance
div Generic block or container Frequently wraps products, cards, sections and repeated records.
span Inline content Often contains prices, labels, statuses or short values.
a Link Usually contains an href that can be followed with a Link selector.
img Image Usually contains image information in src, srcset or related attributes.
button Interactive control May load more records, change variants, open tabs or trigger navigation.
ul, ol, li Lists Common structures for navigation, categories, specifications and repeated items.
table, tr, td, th Tabular data Useful for specifications, comparison tables and structured records.
input, select, option Form controls May represent search fields, filters or product variation controls.

Common HTML attributes

Attribute Example Use when scraping
id id="product-list" Useful when the ID is stable and unique.
class class="product-card featured" The most common way to target groups of similar elements.
href href="/product/123" Contains a link destination.
src src="/images/item.jpg" Common source for images and other embedded resources.
alt alt="Black running shoe" May contain useful image description text.
title title="In stock" Can contain additional metadata not shown as normal text.
data-* data-sku="ABC-123" Often contains stable identifiers, state or structured values.
name, value value="blue" Frequently used by forms, filters and variation controls.

Parent, child and descendant relationships

HTML elements are nested. The surrounding element is the parent, an element directly inside it is a child, and any element further inside it is a descendant.

<div class="product-card">
  <a class="product-link" href="/product/123">
    <span class="product-name">Example product</span>
  </a>
  <span class="price">$19.99</span>
</div>

Here, .product-card is the parent of .product-link and .price. The .product-name element is a descendant of .product-card and a child of .product-link.

This relationship matters because child selectors in a Web Scraper sitemap execute within the context returned by their parent selector.

Repeated HTML structures

Listing pages commonly repeat the same wrapper structure for every record.

<div class="product-card">...first product...</div>
<div class="product-card">...second product...</div>
<div class="product-card">...third product...</div>

An Element selector can target .product-card and return each matching element as a separate parent context. Child selectors such as .product-name and .price then extract values within each product card rather than across the entire page.

CSS selector basics

A CSS selector describes which HTML elements should match. Web Scraper can generate selectors with the point-and-click interface, or you can enter a selector manually.

Pattern Example Matches
Element type div All div elements.
Class .product-card Elements containing the product-card class.
ID #product-list The element with the matching ID.
Element and class a.product-link Link elements with the product-link class.
Multiple classes .product-card.featured Elements that contain both classes.
Descendant .product-card .price Elements with class price anywhere inside .product-card.
Direct child .product-card > .price Elements with class price that are direct children of .product-card.
Adjacent sibling .label + .value The first .value immediately after .label.
General sibling .label ~ .value Matching .value siblings that appear after .label.

Attribute selectors

Attribute selectors are useful when a stable class is unavailable or when the value is stored directly in an HTML attribute.

Selector Meaning
[data-sku] Elements that have a data-sku attribute.
[data-state="active"] Elements whose data-state value exactly equals active.
[href^="/product/"] Elements whose href starts with /product/.
[href$=".pdf"] Elements whose href ends with .pdf.
[class*="product"] Elements whose class attribute contains the text product.

Prefer exact and stable attributes when possible. Partial attribute matching is useful when only part of an attribute value remains consistent.

Combine selector conditions

Selector parts can be combined to make a match more specific.

div.product-card[data-state="active"] a.product-link

This selects a.product-link elements inside active div.product-card elements.

Use the shortest selector that reliably identifies the required elements. Long selectors copied from the full DOM hierarchy are usually harder to maintain.

Target alternative elements with a comma

Separate CSS selectors with a comma when the same field can appear under different markup.

.price, .original-price

This matches elements that satisfy either selector. Matching elements are returned in document order, not in the order the selector expressions are written. If .original-price appears before .price in the HTML, it is matched first even though .price is written first.

See CSS selectors for the full Web Scraper multiple-record and first-record behavior.

Exclude elements with :not()

Use :not() when a stable condition identifies elements that should be excluded.

.product-card:not(.sponsored)

This matches product cards that do not have the sponsored class.

jQuery selector extensions

Web Scraper supports jQuery-style selector extensions in addition to browser CSS selector syntax. For scraping, the most useful extensions are text filtering and descendant filtering.

These expressions are selectors. Do not enter JavaScript or jQuery program code into a Web Scraper selector field.

Select elements by text with :contains()

button:contains("Load more")

This matches button elements containing the text Load more. Text matching is case-sensitive, so use the text as it appears in the rendered page.

You can combine it with normal CSS syntax:

.filters button:contains("In stock")

Select elements that contain another element with :has()

.product-card:has(.price)

This matches product cards that contain an element matching .price.

The selector inside :has() is evaluated as a descendant condition. This is useful when the parent wrapper has no unique class or attribute of its own.

Combine jQuery conditions with :not()

jQuery conditions can be inverted when the element should be selected only when specific content is absent.

.product-card:not(:has(.out-of-stock))

This matches product cards that do not contain an element with the out-of-stock class.

button:not(:contains("Sold out"))

This matches buttons that do not contain the text Sold out.

Chain selector conditions

CSS and jQuery conditions can be combined when a single condition is not specific enough.

.product-card:has(.price):not(:has(.sponsored)) a.product-link

This selects product links from product cards that contain a price and do not contain a sponsored marker.

Keep chained selectors readable. If the expression becomes difficult to understand, consider using an Element selector to establish the parent context and simpler child selectors inside it.

Web Scraper parent selector

Use _parent_ from a child selector when you need to select the exact element returned by its parent Element selector.

For example, an Element selector may return each product wrapper while a child Element Attribute selector uses _parent_ to extract a data-sku attribute from that wrapper itself.

Select inside an iframe

iframe:iframe .selector-within-iframe

The first part selects the iframe element. :iframe enters the iframe document, and the selector after it targets elements inside that document.

Browser and site restrictions can prevent access to some iframe content.

Select inside a shadow root

.shadow-host:shadow-root .selector-within-shadow-root

The first part selects the shadow host. :shadow-root enters the shadow root, and the remaining selector targets content inside it.

Selector context in a sitemap

A selector does not always run against the entire page. Child selectors run within the context returned by their parent selector.

For example, if an Element selector returns individual .product-card wrappers, a child Text selector can simply use:

.price

Web Scraper evaluates .price separately inside each product card. You do not need to repeat the complete page-level selector path.

Recommended selector patterns

Goal Example selector
Select repeated product wrappers .product-card
Select a product link inside a wrapper a.product-link
Select one of two possible price classes .price, .original-price
Select elements with a SKU attribute [data-sku]
Select product links by URL structure a[href^="/product/"]
Select a button by visible text button:contains("Load more")
Select wrappers containing a price .product-card:has(.price)
Exclude wrappers containing a sponsored marker .product-card:not(:has(.sponsored))

Choose selectors that survive website changes

  • Prefer meaningful classes, stable IDs and data-* attributes.
  • Prefer a short selector scoped by a parent Element selector over a long page-level selector.
  • Avoid generated class names or IDs that change between sessions or deployments.
  • Avoid relying on an element's position when a stable class, attribute or relationship is available.
  • Avoid copying a complete selector path from the page root when only part of the path is required.
  • Use Element Preview to confirm that the selector matches the intended elements before scraping.

When a selector returns no elements

  1. Confirm that the target element exists in the rendered DOM.
  2. Test a shorter part of the selector to find where matching stops.
  3. Confirm that the selector is running under the intended parent selector.
  4. Check whether the target appears only after scrolling, pagination, a click or a delay.
  5. Check whether the content is inside an iframe or shadow root.
  6. Check whether classes or attributes change between page loads.

When a selector returns too many elements

  • Add a stable parent class or attribute to narrow the match.
  • Use an Element selector to establish one repeated record as the parent context.
  • Add an attribute condition when elements share the same class but represent different states.
  • Use :has(), :contains() or :not() only when the required distinction is based on descendants or text.

When the wrong value is returned first

When one selector expression matches more than one element, Web Scraper processes matches in document order. Reordering comma-separated selector expressions does not change the order of the matching elements in the HTML.

If you need a different result, make the selector more specific or adjust the selector's multiple-value configuration rather than relying on expression order.

Rendered content and JavaScript

Some websites add or replace elements after JavaScript runs. A selector can be correct but still return no data if it executes before the content exists or if the Cloud driver does not render the required JavaScript.

For dynamic content, identify whether the page requires scrolling, pagination, clicking, a selector delay or the Full Cloud driver. Selector syntax cannot make unavailable content appear.

What this guide deliberately avoids

Web Scraper supports a broad range of browser CSS and jQuery selector syntax, but not every technically possible selector is a good scraping selector. This documentation focuses on stable patterns that are useful across real scraping projects.

  • Avoid positional selectors when a stable class, attribute or structural relationship is available.
  • Avoid selector expressions that depend on generated page structure.
  • Avoid unnecessarily complex chains when parent and child selectors can express the same structure more clearly.

Related