robots.txt and web scraping: what it actually means

A robots.txt file tells compliant automated clients which URL paths a website asks them to access or avoid. It does not technically block a scraper, grant permission to collect or reuse data, or decide whether a scraping project is lawful.

For a real project, treat robots.txt as an important input to a wider decision. Read the rule that applies to your crawler, respect excluded paths, control the total request load, and separately check access controls, terms, data rights and authorised alternatives before scraping at scale.

The short answer: preference, not permission or enforcement

The Robots Exclusion Protocol is a standard way for a service owner to publish crawling rules. A compliant crawler retrieves /robots.txt, identifies the relevant User-agent group and applies its Allow and Disallow rules before requesting other URLs.

Three distinctions prevent most misunderstandings:

RFC 9309 explicitly says the rules are not access authorisation. That distinction works in both directions. A public URL can be disallowed, while an allowed URL can still be subject to terms, privacy obligations, copyright, database rights or a direct agreement.

The file is also public. Listing a path does not keep it secret, so private content still requires real access controls such as authentication.

How to read robots.txt for web scraping

Check the exact origin

Add /robots.txt to the scheme and host you intend to access, for example:

https://shop.example.com/robots.txt

The file belongs at the top-level path and its name is lowercase. Rules for https://www.example.com/ do not automatically govern https://shop.example.com/, a different port or the HTTP version of the site. Check every origin included in the collection plan.

Find the group for your crawler

A group begins with one or more User-agent lines followed by rules. Match the crawler's product token case-insensitively:

• If a named group matches, use it.
• If the same product token has several matching groups, combine their rules.
• If no named group matches, use User-agent: * when it exists.
• If neither a named group nor the wildcard group matches, no robots rules apply to that crawler.

The wildcard group is a fallback. It is not added to a more specific matching group. Do not impersonate a search crawler or another bot to obtain more permissive rules. RFC 9309 expects the product token to correspond to the identification sent by the crawler, and that identification should describe its purpose.

Apply the most specific path rule

Allow and Disallow match from the beginning of the URL path and should be treated as case-sensitive. If several rules match, the longest matching rule wins. If equally specific Allow and Disallow rules conflict, Allow should win.

Consider this example:

User-agent: InventoryBot
Allow: /catalog/
Disallow: /

User-agent: *
Disallow: /checkout/

Sitemap: https://shop.example.com/sitemap.xml

InventoryBot follows its named group, not the wildcard group. /catalog/chairs is allowed under the protocol because /catalog/ is a longer match than /. Every other path is disallowed for that crawler.

An unlisted crawler follows the wildcard group and avoids /checkout/, while other paths have no matching rule. The Sitemap line can help with URL discovery, but it does not override either group or grant permission to collect the listed pages.

Trailing slashes matter. Disallow: /search also matches /searchlight, while Disallow: /search/ targets that directory prefix. RFC 9309 also defines # for comments, * for zero or more characters and $ for the end of a match.

Treat Crawl-delay as a separate signal

Crawl-delay is not defined by RFC 9309, and support is not universal. Google, for example, says its crawler does not support the field. Other crawlers may interpret it differently.

That does not make a clearly published rate preference irrelevant. Record it alongside any limit in the site's crawler policy, API documentation or agreement. Confirm how the chosen implementation measures it, then control aggregate request rate, concurrency and retries across the whole project.

Ten workers each waiting ten seconds can still create roughly one request per second in total. Several individually cautious scheduled jobs can produce the same problem. The guide to ethical request rates explains how to plan the combined traffic budget and respond to server feedback.

Handle missing and unavailable files carefully

RFC 9309 distinguishes between a file that is unavailable and one that is unreachable:

• Successful fetch: follow every parseable applicable rule.
• HTTP 4xx: the file is unavailable, so a crawler may access resources under the protocol. A 404 does not provide contractual or legal permission, while a 401 or 403 should trigger an access review.
• HTTP 5xx or network failure: the file is unreachable, so the crawler must initially assume complete disallow. Pause and check again rather than treating a temporary failure as an opportunity to crawl.

A crawler should normally not use a cached copy for more than 24 hours unless the current file is unreachable. Recurring projects should therefore recheck the file and pause any newly disallowed or uncertain path before the next run.

Keep five separate review layers

The common mistake is asking one file to answer five different questions.

Public visibility does not collapse these layers. Privacy regulators treat scraping of publicly accessible personal information as capable of engaging data-protection law. In the EU, database rights can also be relevant to substantial extraction or repeated and systematic extraction in particular circumstances.

For accessible product facts, an internal review may be straightforward. Personal data, restricted access, substantial copying, redistribution or high-risk use deserves qualified legal review. This article is operational guidance, not advice for a specific project.

Decide whether to proceed, change, ask or stop

First define the exact source and data scope: origins, path patterns, page types, required fields, frequency, volume and intended use. “Scrape the marketplace” is too broad. “Check public product pages daily for price and availability changes” can be evaluated.

Then map representative URLs and live responses to an action:

For each recurring source, retain the checked origin, retrieval time, applicable product token and group, representative path decisions, rate policy, project owner and review outcome. Recheck when the source changes or before materially increasing volume.

When an unrestricted marketplace returns 429

Suppose a marketplace has no matching named group and no wildcard group. Under the protocol, no robots rules apply to the crawler. During a limited pilot, however, the site begins returning 429 Too Many Requests.

The job should pause, honour any Retry-After value and resume only at a materially lower aggregate rate. The absence of a robots restriction does not cancel live server feedback. Sending the same load through additional proxies would change the route, not reduce the load or establish permission.

Validate the pilot, not just the status code

Run a small sample covering ordinary pages, legitimate empty results, layout variants and later pagination. Define pause conditions before scaling, including robots changes, 403, 429, repeated 5xx, CAPTCHAs, unexpected login pages, rising latency, implausible record counts or falling required-field completion.

A response can return 200 OK while containing a challenge, login shell or empty template. The 200 OK but no data diagnostic explains how to validate page markers and extracted records rather than trusting HTTP success alone.

Use Web Scraper within a reviewed scope

Once the source and scope remain supportable, use the free Web Scraper browser extension to build and test a sitemap on representative accessible pages. This separates two questions that are often confused: whether the crawler should request a page, and whether the selectors return a correct record from it.

If the site publishes page lists, the Sitemap.xml link selector can discover sitemap locations listed in robots.txt and filter the URLs to relevant page types. Treat that as efficient discovery, not approval.

After the pilot passes, Web Scraper Cloud can schedule the tested sitemap with an appropriate driver, request interval and page-load delay. Use Fast when the required data is present in raw HTML and FullJS when JavaScript or permitted interaction must create the required page state. Configure data-quality controls for record count, failed and empty pages, and required-field completion.

These controls improve reliability within a reviewed workflow. They do not make a restricted route authorised, and Web Scraper should not be assumed to turn robots.txt into a legal or compliance decision. It is strongest for repeatable datasets from accessible public e-commerce sites, marketplaces, job boards, directories and real estate pages. Social platforms, LinkedIn, large behind-login projects and sources requiring restriction bypass are not its default fit.

If the project passes the source, scope and path review, build and validate a small sitemap first. Automate only the tested paths and keep the robots check, request policy and dataset acceptance rules with the production configuration.