Blog

DataDome’s report published on 15 October 2026 recorded a 4.7× rise in agentic browser requests across its network in September.

For web scraping teams, this raises a practical question: how does a website distinguish an assistant checking a product from a search crawler indexing it or a company collecting its price every morning? Those activities can overlap. Sites need evidence about the operator and its behaviour, then a policy for the requested activity.

Your scraper can return a different price because it receives a different offer, extracts the wrong price from the page, or changes a correct value during processing. Location, saved preferences, login state and selected variants can all affect the comparison.

Start with one mismatched product and find where the two results first diverge. That tells you what to fix before you change proxies, selectors or delays across an entire catalogue.

Reddit plans to test login restrictions in selected markets outside the US in Q4 2026. Announced on 29 October 2026, the move comes as the platform reports falling search referrals and tries to turn more visitors into regular, signed-in users.

For teams collecting public web data, the wider risk is that fewer referral visits could give websites another reason to restrict access. Content can remain useful to an AI search service even when fewer readers visit its source.

A 405 Method Not Allowed response normally means that the requested resource does not support the HTTP method used. During web scraping, a security layer can also return 405 before the request reaches the page. AWS WAF, for example, uses this status for its CAPTCHA action.

The page opening in your browser does not settle which explanation applies. Your browser may send a different request, carry accepted session state or complete a verification flow. Compare the actual requests and responses before changing the scraper.

For recurring product catalogues, marketplace listings, property data and business directories, Web Scraper Cloud is our best-value starting point. It combines managed anti-bot handling with structured extraction, scheduled collection and delivery in one platform.

Residential proxy pricing is one of the first things to compare. A low subscription price can become a substantial operating bill when a protected scraping job transfers hundreds of gigabytes. This guide compares eight tools by their access capabilities, billing models and the work needed to deliver a usable dataset.

The best web scraping tools for niche websites let you define the extraction when a ready-made scraper does not cover your source or required fields. For operations teams building recurring structured datasets without writing scraper code, Web Scraper is our recommended starting point.

Think specialist supplier catalogues, regional retailers, industry directories and smaller job boards. Octoparse, ParseHub, Browse AI and Apify also offer custom setup routes. The useful comparison is what your team can configure, validate and maintain after a template search comes up empty.

Researchers published material about a fictional “Cactus Gravity Protocol” and saw it cited by eight of ten tested AI search platforms, including ChatGPT and Perplexity, within seven days. For businesses using AI to investigate markets or suppliers, the result raises a practical question: what does a citation actually prove?

Cloudflare introduced Disallow AI Training on 15 September 2026, giving website owners a way to refuse model training while keeping supported search crawlers on their sites. It combines published preferences with selective crawler blocking, rather than treating every AI-related request alike.

The setting preserves a route to search discovery, with support that varies by operator. It does not guarantee indexing, rankings or referrals.

AI browser agents can navigate unfamiliar websites, collect information and return a useful comparison. Their results still need checking because successful navigation, accurate individual values and a correct business recommendation are separate achievements.

Before an output informs a purchase, report or recurring dataset, compare its evidence with the original requirements. A confident completion message cannot establish that the agent found every required source, compared equivalent products or applied the conditions that make its recommendation valid.

The European Data Protection Board adopted draft AI scraping guidelines for consultation on 7 July 2026. They explain how existing GDPR duties apply when businesses collect web data to develop generative AI.

As of 16 October 2026, Guidelines 03/2026 remain open for consultation, with comments due by 30 October 2026. That is the feedback deadline. Businesses already processing personal data must assess their activities under the existing GDPR framework. The practical recommendations below explain how to turn that assessment into collection and purchasing decisions.

Businesses are scraping ChatGPT and Perplexity to study the answers they produce: which products they recommend, how they describe companies and which pages they cite. AI services have become another source of competitive intelligence, alongside the websites they help people discover.

To scrape restaurant menus and prices, select each dish's container, then extract its name and price from inside that container. Keep the branch, menu period and portion size with the result so that each price describes the right offer.

With Web Scraper, you can build and test this structure in the browser extension, export a sample, then move the validated sitemap to Cloud for recurring collection.

To scrape a website using its XML sitemap, use the published URLs to find the pages you need, then extract data from those pages. In Web Scraper, the Sitemap.xml Link selector handles URL discovery and navigation, while child selectors collect fields such as product names, prices and SKUs.

For a product catalogue, this can save you from building navigation through categories and pagination. The workflow below uses an illustrative shop to show how to configure the scraper, check the resulting records and automate a validated setup.

To extract JSON-LD structured data from a website, find its script[type="application/ld+json"] elements and collect their contents. In Web Scraper, an HTML selector captures these blocks. Web Scraper Cloud can then isolate simple fields from consistent markup and automate repeat collection.

For product research or price monitoring, first check that the markup describes the product and offer you actually need. A neatly structured block can still contain a default variant, a price range or unrelated entities.

A page can load successfully and still show the wrong data. A product price may depend on the selected size, a directory may need a location, and a job board may reveal more listings only after a click. Executing actions during web scraping means reproducing the steps that create the specific page state your dataset requires.

The useful sequence is action → expected state → readiness → extraction → record check. A click that succeeds without producing the intended records is not a successful scraping step.