How to turn a website into data for an AI assistant

Turning a website into data for an AI assistant requires more than saving every page as text. You need to define what the assistant should answer, extract one clean and attributable record per page, validate the resulting dataset, and only then pass accepted content to chunking and indexing.

This guide applies that workflow to a public documentation website used by a support assistant. The goal is a current, traceable source dataset, not an end-to-end RAG platform.

Define the assistant’s evidence before scraping

Start with the questions the assistant should answer. A crawler cannot tell you whether a dataset is suitable if the expected answers have not been defined.

Suppose a software company wants its support assistant to answer questions about authentication, API behaviour and recent product changes. Its initial test set might include:

• How do I configure single sign-on?
• Which authentication method does the API use?
• Did the latest release change this setting?
• Which documentation page supports the answer?

Add questions the source should not answer, such as an individual customer’s account status or an unpublished product roadmap. The expected response may be an admission that the information is unavailable or a route to human support.

This establishes four parts of the data contract:

1. the approved source and sections;
2. the page types containing the required evidence;
3. the metadata needed to filter and cite that evidence; and
4. the maximum acceptable age of the source data.

Microsoft’s RAG preparation guidance similarly recommends gathering representative content and test queries together. If no source passage can answer a test question, better chunking will not create the missing evidence.

Map the website by page family

Do not begin by following every internal link. First divide the source into page families with different content roles and layouts.

For docs.example.com, the plan might look like this:

Page family	Example pattern	Decision	Content to retain
Guides	/guides/*	Include	Title, article body, headings, lists, code and source URL
API reference	/api/*	Include	Method, path, descriptions, tables, code and source URL
Release notes	/releases/*	Include	Version, date, change descriptions and source URL
FAQ pages	/faq/*	Include if relevant	Question, answer, section and source URL
Search and tag pages	/search, /tags/*	Exclude as documents	Use only for discovery when needed
Account pages	/account/*	Exclude	User-specific content is outside this project
Old product versions	/v1/*	Decide explicitly	Retain only with version metadata and a retrieval rule
Other languages	/fr/*, /de/*	Decide explicitly	Retain only with reliable language metadata

An XML sitemap can seed the URL inventory, while navigation can expose additional sections. Neither proves that coverage is complete. Compare candidate URLs with the website’s visible structure and track counts by page family.

In a Web Scraper sitemap, start URLs and navigation selectors determine which pages are reached, while data selectors define the output fields. Use separate branches or sitemaps when page families have meaningfully different templates, refresh policies or failure risks, while retaining one downstream schema.

Define one page record before building selectors

The scraper should produce a page dataset, not an accidental collection of DOM fragments. Define the record before configuring extraction.

Field	Purpose	Created by	Required?
source_url	Opens the page supporting the record	Extraction	Yes
identity_url	Normalised URL used for identity matching	Ingestion	Yes
declared_canonical_url	Canonical URL published by the page, when present	Extraction	Optional
title	Human-readable page title	Extraction	Yes
content_html	Selected article structure for cleaning and chunking	Extraction	Recommended
content_text	Clean plain-text representation	Extraction or ingestion	Optional
page_type	Guide, reference, release note or FAQ	Extraction or rule	Yes
language	Prevents retrieval from the wrong locale	Extraction or rule	When relevant
product_version	Distinguishes current and historical guidance	Extraction or rule	When relevant
source_updated_at	Date claimed by the publisher	Extraction	Optional
retrieved_at	Time this representation was collected	Parser or ingestion	Yes
document_id	Stable identity across refreshes	Ingestion	Yes
content_hash	Detects changes to cleaned content	Ingestion	Yes
schema_version	Identifies the record contract	Ingestion	Yes
validation_status	Prevents rejected records reaching the index	Ingestion	Yes

Keep source_updated_at and retrieved_at separate. A collection timestamp does not prove when the publisher changed a page. If no trustworthy source date exists, leave it empty and retain the retrieval time.

The W3C’s Data on the Web Best Practices recommends persistent identifiers, provenance, quality information and version indicators. These fields make the source dataset easier to trace, filter and refresh.

Build and test the extraction in the browser

Use the Web Scraper browser extension to build the sitemap against the live website. Start with a deliberately varied set: a short guide, a long guide, an API table, a page containing several code blocks, a release note and an older page without an update date.

A simple selector tree might follow this pattern:

_root
└── documentation_links
    ├── title
    ├── declared_canonical_url
    ├── product_version
    ├── source_updated_at
    └── content_html

documentation_links follows approved article links from an index page. Its child selectors extract fields from each destination article.

For every page family:

1. select the page title separately;
2. retain the destination URL;
3. select only the main article container;
4. capture version, language and update metadata when available; and
5. confirm that every family produces the required columns.

Use the simplest rendering route that returns the required content. Web Scraper’s Fast driver reads returned HTML, while FullJS executes JavaScript and supports interaction-dependent workflows. Use FullJS when content or navigation appears only after rendering or a known interaction, not merely because the website uses JavaScript somewhere.

Before automation, preview and validate the sitemap across different layouts, later navigation pages and content revealed by interaction. Then run a limited scrape and inspect the actual export. One correct guide does not prove that API pages, release notes and later pagination states work.

Extract the article, not the website shell

Selecting the entire body can add navigation, a table of contents, cookie notices, feedback widgets, related links and footer text to every record. That repeated furniture can outweigh the unique content, create false changes across the corpus and appear in retrieved passages as if it were part of the article.

Select the narrowest stable container that contains the complete article. Retain meaningful structure such as:

• heading order;
• numbered procedures;
• parameter and compatibility tables;
• warnings and prerequisites;
• command examples and code blocks; and
• version or platform notes.

Capture the article with an HTML selector when that structure matters. The ingestion layer can sanitise the selected HTML and derive plain text without rediscovering the article boundary inside the complete website shell.

If the correct raw value needs predictable processing, such as whitespace normalisation, a derived column or a scrape timestamp, use Web Scraper Cloud Parser. If the sitemap selected the wrong source content, repair the selector rather than trying to clean the wrong value afterwards.

Aim for one output record per logical page. Splitting every paragraph during extraction ties source acquisition to one retrieval strategy and makes identity, attribution and change comparison harder to manage.

Add stable identity and provenance downstream

Keep these concepts separate:

• document_id identifies the logical page across runs.
• content_hash represents the cleaned content collected in one run.
• version_id can combine the document identity and content hash.
• chunk_id identifies one derived retrieval unit.

Do not derive document identity from a page title, export position or chunk number. Titles can change, multiple pages can share one title and crawl order is not stable.

A practical identity process is:

1. record the final source URL;
2. capture the page’s declared canonical URL when one exists;
3. verify that the canonical belongs to the approved source and represents the expected page;
4. remove fragments and explicitly recognised tracking parameters;
5. retain parameters that change language, version, region or content; and
6. create document_id from a collection namespace and the resulting identity_url.

A canonical tag is useful evidence, not an instruction to merge pages automatically. An incorrect or overly broad canonical can collapse distinct documents into one identity.

One enriched page record might look like this:

{
  "document_id": "docs-example-en-guides-sso",
  "source_url": "https://docs.example.com/guides/sso?ref=nav",
  "identity_url": "https://docs.example.com/guides/sso",
  "declared_canonical_url": "https://docs.example.com/guides/sso",
  "page_type": "guide",
  "title": "Configure SSO",
  "language": "en",
  "product_version": "current",
  "source_updated_at": null,
  "retrieved_at": "2026-09-07T06:00:11Z",
  "content_html": "<h2>Requirements</h2><p>...</p>",
  "content_hash": "sha256:...",
  "schema_version": "docs-page-v1",
  "validation_status": "accepted"
}

This is a conceptual ingestion record, not the Web Scraper Cloud API schema. Web Scraper extracts the configured source fields and can add the scrape time. The ingestion service creates identifiers, hashes and validation state.

Validate the dataset before indexing it

A completed scraping job is not automatically a correct dataset. Validate it at three levels.

Level	Acceptance checks
Page	Expected host and page type, required fields, correct language and version, meaningful content, no login or challenge text
Dataset	Unique IDs, plausible counts by page family, required-field completion, duplicate-content checks, failed and empty pages, comparison with the previous accepted run
Retrieval	Known-answer questions retrieve the supporting section, citations open the correct URL, version filters work and unanswerable questions remain unsupported

Web Scraper Cloud data quality controls can check minimum record count, failed-page percentage, empty-page percentage and required-field completion. Establish thresholds from representative accepted jobs and allow for normal source variation. A quality failure remains separate from the scraping job’s terminal status.

Do not rely on HTTP status alone. A request can succeed while returning a consent screen, challenge, empty application shell or wrong locale. The guide to diagnosing 200 OK responses with no usable data covers that failure pattern in more detail.

Before automating the entire source, require a pilot batch to prove that all included page families are reachable, required structure survives extraction, every record has usable identity and provenance, the JSON reaches the ingestion service, and a second run does not create unexplained duplicates.

Hand accepted pages to chunking and indexing

Web Scraper Cloud can deliver newline-delimited JSON exports, while its API can download job data as JSON. The data export documentation lists the available formats and delivery routes.

Keep each accepted page record as a recoverable source artefact. Then create chunks in the ingestion layer:

1. sanitise the selected HTML while retaining meaningful structure;
2. split first at heading boundaries;
3. merge sections that are too small to carry useful context;
4. split oversized sections at paragraph or sentence boundaries;
5. add overlap only when testing shows that a boundary loses necessary context; and
6. copy provenance and filtering metadata to every chunk.

Each chunk should retain at least the document_id, source_url, page title, heading path, page type, language, product version, retrieval time and content version. The title and heading path can also be prepended to the chunk text so it remains understandable when retrieved alone.

There is no universal correct chunk size. Microsoft’s chunking guidance explains why document structure and expected queries should drive the choice. A release note, API table and long troubleshooting guide may need different rules.

Web Scraper owns the configured extraction and delivery layer. The downstream system owns HTML sanitisation, document identity, change detection, chunking, embeddings, index updates, retrieval and answer generation.

Refresh the data without corrupting the index

Choose the refresh interval from each page family’s change rate and the maximum acceptable answer age. Release notes may justify more frequent collection than stable reference pages.

Once the sitemap has been tested, Web Scraper Cloud scheduling can run it repeatedly. For an event-driven pipeline, webhook notifications report when a job is finished, stopped or failed. The webhook is a completion signal, not the dataset or a quality certificate.

Compare each accepted dataset with the previous accepted version:

State	Evidence	Downstream action
New	New document_id	Clean, chunk, embed and insert
Changed	Same ID, different content_hash	Replace that document’s derived chunks
Unchanged	Same ID and hash	Keep the existing indexed content
Missing	Previous ID is absent	Hold for confirmation before deletion
Invalid	Schema or content check fails	Quarantine and investigate
Failed page	Retrieval or rendering failed	Retain the last accepted version under the freshness policy

The missing state needs particular care. If navigation breaks halfway through a run, hundreds of absent records do not represent hundreds of deleted pages. Propagate removals only after the new run passes coverage checks and the absence is credible. Depending on the source, confirmation may require a recognised removal page, another successful run or a grace period.

A safe production sequence is:

1. complete the scraping job;
2. download or receive the complete dataset;
3. validate its schema, fields and coverage;
4. compare it with the last accepted version;
5. process new and changed documents in staging;
6. run the retrieval test set; and
7. promote the accepted update.

This prevents a plausible but incomplete dataset from silently replacing good data.

Where Web Scraper fits

Web Scraper fits the repeatable acquisition layer for known, accessible public sources. The browser extension builds and tests the sitemap. Cloud adds scheduled execution, Fast and FullJS drivers, job monitoring, data quality controls, webhooks and complete-dataset delivery.

It does not define your document identity, choose chunk boundaries, create embeddings or operate the search index. Those responsibilities remain in the ingestion and application layer, as shown in the broader web data for RAG systems and AI agents workflow.

This is not a default workflow for social platforms, LinkedIn, large projects behind login or content whose collection and reuse have not been approved. Review crawler rules, website terms, privacy, copyright and applicable obligations separately. No scraping tool can guarantee compatibility with every website, so test representative pages before designing the production pipeline.

Frequently asked questions

Should I scrape the entire website for an AI assistant?

Usually not. Include the page families that answer the assistant’s approved questions. Exclude search pages, duplicate archives, account areas and irrelevant marketing content. Broader coverage can add noise as well as information.

Is JSON enough for an AI assistant?

JSON is a delivery format, not a guarantee of useful content. The records still need clean article content, stable identity, provenance and validation before the ingestion layer chunks and indexes them.

Should chunking happen in the scraper?

Keep scraping and chunking separate unless the downstream system specifically requires pre-chunked records. Retaining one accepted record per page makes it easier to change chunking rules, rebuild an index and compare source versions without scraping again.

How often should website data be refreshed?

Set the schedule from the source’s actual change rate and the maximum staleness the use case can tolerate. Always validate a new run before it updates the live index.

Can Web Scraper turn every website into AI-ready data?

No. Test representative pages in the extension, confirm that the source is appropriate for collection, and validate the exported dataset before automating it.

Ready to build the acquisition layer? Start with one representative page from each source family in the Web Scraper browser extension, validate the page dataset, and move the tested sitemap to Cloud when recurring delivery is required.