Building a fresh web data pipeline for RAG

A fresh RAG data pipeline at a glance

Retrieval-augmented generation, or RAG, retrieves external evidence and supplies it to a generative model at answer time. This makes the model's usable knowledge easier to update without retraining it. It does not make that knowledge current automatically.

A production RAG data pipeline is fresh when the newest valid information is queryable, and obsolete versions are no longer eligible for ordinary retrieval, within a defined time objective. Freshness is therefore an end-to-end property of discovery, collection, validation, versioning, indexing and retrieval - not a scheduler setting.

This guide is for developers, data teams and engineering leaders who already have, or are selecting, a RAG application and retrieval store but need a dependable way to supply them with changing public web data.

Web pages change without notice. JavaScript can hide content from a raw-HTML collector, extraction can fail despite a successful request, and an index can retain an old document after a crawler finds its replacement.

The original RAG paper made external knowledge easier to update by separating it from the model's parameters, but did not make freshness automatic. Research on fast-changing knowledge found that current search evidence improved benchmark answers, while its quantity and ordering still affected correctness. Retrieval creates a place for fresh evidence; the pipeline must put the right version there.

The central rule is therefore: a completed crawl is not a fresh RAG corpus. Freshness is achieved only when a validated source change has propagated into retrieval and the obsolete version can no longer win.

Four short contracts make that rule operational:

• The source contract defines approved content, acquisition method and expected volatility.
• The document contract defines identity, required fields, timestamps, versions and provenance.
• The publish contract defines what a candidate must prove before becoming current.
• The freshness contract defines maximum propagation delay and acceptable obsolete-version exposure.

"The index refreshes every hour" is a schedule, not a freshness contract. A useful objective describes the required state, time limit and proof.

An hourly job can still serve old information if it misses rendered text, fails to remove a deleted page or leaves old chunks active. A green scheduled task is not evidence that the answer changed.

Freshness is a data state, not a schedule

Before choosing tools, define "fresh" for the application. Product documentation may tolerate a longer delay than availability or pricing data. One schedule can waste resources on the first and produce stale answers from the second.

Track at least four timestamps for every document:

• Source modified time: When the publisher says the content changed, if a trustworthy value exists.
• Retrieved time: When the collector last obtained a valid representation.
• Content changed time: When the normalised content hash last changed.
• Queryable time: When the current version was verified as available to retrieval.

Do not collapse these into updated_at. A page can be observed today but unchanged for months; another may have changed this morning but remain unindexed because embedding failed. One timestamp cannot describe both without becoming impressively unhelpful.

Measurements using source_modified_at require a trustworthy publisher value. Otherwise, observation age and ingestion lag are the honest measures: how recently the pipeline checked and how quickly a valid observation became queryable.

The appropriate collection architecture depends on the question and source:

Scheduled collection is usually the dependable base. Events can reduce delay, while reconciliation crawls catch what they miss. Live retrieval should be an explicit route, not an excuse to leave the maintained corpus untended.

Building the web data pipeline for RAG

1. Define the corpus and source contracts

Start with a source registry rather than a crawler configuration. For each domain or section, record its authority, permitted use, discovery method, rendering requirement, expected change rate, owner and freshness objective.

A documentation site might classify release notes as high-volatility content, reference documentation as moderately changing and archived material as low-volatility. This is more useful than assigning the entire domain one interval and hoping the arithmetic feels fair.

Identify which source wins when pages disagree. The newest page is not necessarily the most authoritative: official documentation and a forum answer can discuss the same topic with very different evidential weight.

Exclude search pages, navigation, tag archives, translated duplicates and low-value user-generated sections where they add no answer value. More text is not automatically more knowledge. Sometimes it is merely more footer.

Define required fields by page type. A product page may need an identifier, price, currency and availability; a policy may need its title, effective date, headings and body. These become validation rules before the index accepts an empty document.

2. Discover URLs and assign stable document IDs

URL discovery can combine XML sitemaps, feeds, category pages, internal links, APIs and seed lists. Use feeds and APIs for reliable events or identifiers, then scrape the browser-facing page when it contains additional required content. The choice is often field-specific, as explained in our web scraping versus API guide.

Sitemap <lastmod> values are useful hints only when the publisher maintains them accurately. Google's sitemap guidance says the value should represent the last significant page update and explains that it uses the value when it is consistently verifiable. Treat an unreliable lastmod as a scheduling suggestion, not a content checksum.

Compare the latest valid canonical inventory with the current candidates only after the new run passes coverage checks. A crawl that stops after page one does not prove that the rest of the catalogue was deleted.

Each logical document needs an identity independent of its text. Prefer a publisher's stable entity ID; otherwise derive document_id from a source namespace and validated canonical key.

The timestamps above are illustrative test data, not claims about a live source.

Canonicalise deliberately. Remove tracking parameters and map known aliases, but retain parameters that select different content, languages or regions. Untidy URLs can still represent different documents.

3. Collect the content the user can actually see

Use the lightest method that returns the required content correctly. Static pages often work with raw HTML; content populated after load, clicks or scrolling may require browser execution.

This is where collection success and dataset correctness commonly diverge. A server can return status 200 while the article body is absent from the initial HTML. The fetch succeeded; the document did not.

A representative pilot should include:

• Static and dynamic pages: Test both ordinary pages and awkward templates that require rendering or interaction.
• Pagination and deep content: Confirm that discovery reaches pages beyond the first category screen.
• Regional and device variants: Check whether location, cookies or viewport change the information shown.
• Failure states: Detect consent screens, soft 404s, login walls, rate-limit pages and bot challenges as failures, not knowledge.

Record the acquisition mode. Raw HTML, structured data and rendered text can disagree because of caching, regional variants or experiments. The pipeline should explain which representation it saw.

The Web Scraper browser extension can build sitemaps visually and preview fields against rendered pages. The same sitemap can run in Web Scraper Cloud with the raw-HTML Fast driver or JavaScript-capable Full driver, as documented in the Cloud overview.

4. Preserve raw and normalised layers

Keep a raw or near-raw snapshot long enough to debug extraction changes and reprocess important documents. Once text has been stripped, chunked and embedded, reconstructing why a sentence disappeared becomes unnecessarily archaeological.

The normalised layer should contain meaningful headings, text, selected tables, canonical URL and provenance. Remove scripts, menus, repeated headers and unrelated recommendations. Normalise whitespace and encoding before hashing so cosmetic differences do not trigger work.

Retain heading paths, table headers, list relationships and page type rather than flattening everything. Preserve dates, language, region and identifiers as typed metadata instead of burying them in text.

Version normalisation rules. A parser update can change the content hash without a source change; extraction and schema versions let the pipeline distinguish the two.

Content collected from the public web is untrusted input. OWASP's guidance on indirect prompt injection specifically describes instructions arriving through external sources such as websites or files. Do not treat retrieved page text as application instructions. Separate instructions from evidence, classify source trust, restrict connected tools to the least privilege required, validate outputs and include adversarial source pages in security testing. Removing scripts is sensible web hygiene, but malicious instructions can also appear in ordinary visible text.

5. Detect meaningful changes and deletions

Freshness usually depends on incremental indexing. Revisit pages according to the source objective, but only process a new version when its meaningful normalised representation changes.

Use several signals in order:

• Publisher signals: Reliable feeds, API events and sitemap modification dates can prioritise URLs.
• HTTP validators: ETag or Last-Modified conditional requests can avoid transferring an unchanged representation. A 304 Not Modified response allows reuse under HTTP caching rules.
• Normalised content hash: Hash the extracted document after boilerplate removal. This remains the decisive change signal when publisher metadata is missing or unreliable.
• Discovery diff: Compare the current complete canonical inventory with the previous complete successful inventory to identify additions and possible removals.

A source event or lastmod change should enqueue collection and validation, not mutate the corpus. The resulting valid observation determines whether a version is published.

Deletion needs its own policy. A confirmed 404 or 410 can be a strong signal; a timeout, CAPTCHA or single failure is not. Retry uncertain pages and create a tombstone only after confirmation. Otherwise a short outage can make the assistant fresh and useless.

A tombstone records why and when a document was retired and prevents a delayed retry from restoring it casually.

6. Use stable identities, then chunk and index with provenance

Random IDs and append-only writes create duplicate truth. Use a four-level identity hierarchy instead:

This is a pattern, not a universal schema. The four IDs respectively identify the configured source, logical document, normalised version and retrieval unit produced under a known chunking policy.

The hierarchy also clarifies three different comparisons:

• An exact content hash asks whether this normalised document changed.
• Near-duplicate detection asks whether two records substantially overlap.
• Semantic similarity asks whether two pieces of text concern similar meaning.

Semantic similarity must not decide identity. Two versions of a returns policy should be similar even when one is obsolete. Using similarity to merge them preserves the conflict that versioning should remove.

Chunk after normalisation. Headings, clauses, product records and table groups usually preserve meaning better than arbitrary character windows. Use overlap only where testing shows a benefit.

Every chunk should inherit enough metadata to answer three operational questions: Where did this come from? Which document version produced it? Is it eligible for current retrieval?

• Identity: Source, document, version and chunk IDs.
• Provenance: Canonical URL, source domain, page title, heading path and retrieval time.
• Freshness: Source modified time, content changed time and indexed time.
• Controls: Language, region, content type, authority tier and access scope.

Stable boundaries may allow selective re-embedding. Rebuild when a layout, schema or chunking change could leave mismatched context or orphaned chunks, and always store the chunker version. A vendor default is a starting point, not a law of information retrieval.

7. Publish new versions and remove old ones safely

An upsert is a storage operation, not a publication protocol. A safe publication should follow seven steps:

1. Produce and validate the snapshot: Confirm that the candidate document is complete, correctly typed and derived from an approved source observation.
2. Generate the expected manifest: List the complete set of chunk IDs and required metadata for the new version.
3. Stage the candidate: Write every new or changed chunk with its version_id, keeping it outside the current retrieval view.
4. Verify queryability: Wait for the write to complete, then use count checks and smoke queries to prove that the candidate can actually be searched.
5. Promote the version: Move the validated candidate into the current retrieval scope through an atomic document update, active-version pointer or controlled index cutover.
6. Retire surplus chunks: Remove or deactivate chunks from the previous version that are not part of the active manifest.
7. Confirm the current view: Verify that ordinary current-state queries can retrieve the new version and cannot retrieve the retired one.

The fourth step matters because a successful write acknowledgement does not always mean the data is searchable. Pinecone's documented limitations, for example, state that the service is eventually consistent and that recently upserted records can take a short time to become available to queries. Treat acknowledgement as the start of publication verification, not its completion.

Frequent updates often suit controlled per-document replacement; larger pipeline migrations may justify a versioned or blue-green index. In either case, gate record coverage, metadata, duplicates, empty content, chunk-count changes, embedding failures and deletion volume. A run producing dozens of documents where thousands are expected should not be published.

Worked example: when 30 days becomes 14

Consider a retailer whose returns page says customers may return an item within 30 days. The active RAG document has the stable ID policy:returns, a version derived from its normalised content and several chunks eligible for current retrieval.

The retailer changes the policy to 14 days. A sitemap signal may prioritise the URL, but the pipeline must still collect and validate what the page now says.

The update moves through these states:

1. Collect: Fetch the correct representation, including browser rendering if JavaScript inserts the policy text.
2. Normalise: Remove navigation and page furniture, preserve the policy structure and produce the same stable document_id.
3. Detect: Compare the new normalised content hash with the active version and create a new version_id.
4. Validate: Confirm that the required policy content is present and that the observation is not a consent screen, challenge page or empty template.
5. Stage: Create and embed the complete new chunk manifest as an inactive candidate. The 30-day version remains current while this happens.
6. Verify: Run a staging query such as “How long do I have to return an item?” and confirm that the 14-day passage is searchable with the correct source metadata.
7. Promote and retire: Make the new version current, remove the 30-day chunks from the current scope and verify both actions in production.

If validation or queryability checks fail, the 30-day version stays active and the candidate is quarantined. Replacing it with malformed content would merely make the error newer.

The naive alternative is to insert the new chunks with random IDs and leave the old chunks untouched. Both policies then exist in the index, and semantic similarity decides which one the user sees. The vector database has no moral objection to serving last month's policy. It has merely found it relevant.

8. Measure the complete path to retrieval

Monitoring must continue beyond the crawler. The pipeline is only working when a changed fact has passed validation, become queryable from the current version and displaced the obsolete representation.

Turn real changes into publication tests. Query for an updated policy, removed product or new release note and inspect the returned version and source. Generic search success proves little.

Break metrics down by source, page type, acquisition mode and pipeline version. Aggregate success can conceal a broken template. Compare coverage, document length, field population and chunk counts with the last known-good baseline.

Retrieval rules for historical questions and generation-level evaluation require additional design beyond this ingestion pipeline.

Using Web Scraper Cloud as the collection layer

As of August 2026, Web Scraper Cloud can provide the repeatable collection and automation layer of this architecture. It does not replace the downstream document modelling, versioning, embedding, index publication and application-specific retrieval components.

A practical workflow is:

1. Build the sitemap: Configure navigation and extraction in the browser extension, including links, pagination, dynamic content and the fields required for provenance.
2. Select the execution method: Use the Fast driver when raw HTML contains the required data, or the Full driver for JavaScript-rendered and interactive pages.
3. Run collection: Use the Scheduler for recurring checks or the Web Scraper Cloud API to launch jobs against an existing sitemap.
4. Handle completion asynchronously: A webhook can notify your endpoint when a job finishes, stops or fails. Acknowledge promptly, queue the import and make processing idempotent because webhook delivery can be retried.
5. Apply a collection quality gate: Configure data-quality controls for minimum record counts, failed and empty page percentages and required field population.
6. Continue downstream: Store the source snapshot, normalise the data, compare hashes, confirm removals, create versions, chunk changed documents, embed them and run the publication sequence.

The Cloud parser can perform repeatable post-processing such as stripping HTML, removing whitespace, applying regular expressions and creating virtual columns. Keep application-specific document modelling, identity and chunking in the RAG ingestion layer, where those decisions can be versioned and tested alongside publication.

Web Scraper Cloud data-quality controls can detect suspicious output, but they do not repair a changed selector or prove that a candidate index is searchable. That boundary is important: collection infrastructure can report that the latest dataset looks wrong; the publish contract must prevent it from replacing a known-good current version.

Data governance and source safety

A technically reachable page is not automatically an approved source. Record the applicable terms, licences, internal permissions, retention requirements and intended uses for each source. If personal data, copyrighted material or restricted content is involved, obtain advice appropriate to the jurisdiction and use case.

Respect source capacity with caching, request pacing and targeted refreshes. The Robots Exclusion Protocol is an important crawl-coordination standard, but it is not access authorisation. It belongs in the source review, not in place of one. Our article on scraping public data responsibly provides further general context.

Provenance should survive the complete pipeline. Retain the canonical URL, source identifier, retrieval time, content hash, processing versions and access scope with every document. This supports citations, removal requests, audits and targeted reprocessing when extraction logic changes.

Production checklist

• Define every source: Record authority, permission basis, owner, discovery method, acquisition mode and freshness objective.
• Write the four contracts: Make source, document, publish and freshness requirements explicit.
• Test representative pages: Include static, dynamic, deep, regional and failure-state pages rather than only ideal URLs.
• Preserve evidence: Retain source snapshots, provenance and processing versions long enough to explain an extraction change.
• Use the identity hierarchy: Keep source and document IDs stable, version meaningful content states and make chunk generation deterministic.
• Validate before promotion: Check coverage, field population, duplicates, chunk changes, timestamps and embedding failures while the last known-good version remains active.
• Handle removals conservatively: Retire every chunk for a confirmed deletion, but never infer mass deletion from a partial or failed crawl.
• Verify publication: Confirm that the complete new version is searchable before promotion and that superseded chunks leave the current scope afterwards.
• Monitor the full path: Alert on observation age, propagation lag, publication failures, retirement failures and invalid source links.
• Protect the application: Treat retrieved content as untrusted and limit the permissions available to systems that consume it.

Frequently asked questions

How often should a RAG data pipeline refresh web content?

There is no universal interval. Set cadence by source volatility, business impact and the full propagation budget. Use reliable change events to prioritise work, scheduled collection as a dependable base and periodic reconciliation to find missed additions or removals. Measure the age of queryable evidence, not only how often a job starts.

Should every document be re-embedded on every crawl?

No. Record the successful observation, normalise its content and compare the resulting hash with the current version. Skip downstream processing when the meaningful representation is unchanged. Stable structural boundaries may support changed-chunk updates, but rebuild the document when its schema, extraction logic or chunking policy changes in a way that could leave inconsistent context.

Can sitemap lastmod values drive incremental indexing?

They can prioritise collection when the publisher maintains them accurately, but they should not be the only change signal. Confirm meaningful changes with the collected and normalised representation, and run periodic reconciliation to find pages that unreliable metadata missed.

How do I prevent duplicate chunks in a vector database?

Use deterministic source, document, version and chunk IDs; stage a complete expected chunk manifest; and make retries idempotent. Promote the validated replacement, then retire chunks that do not belong to the new current version. Random IDs with append-only writes make duplicate current truth almost inevitable.

How should deleted pages be handled in RAG?

Create a tombstone from an authoritative removal event, a comparison between complete successful inventories or a conservative repeated-absence rule with confirmation. Do not interpret a partial crawl, bot challenge or temporary source failure as proof that every missing page was deleted.

Does a recently updated index guarantee a current answer?

No. The candidate may not yet be queryable, an old version may still be active or the wrong source may be selected. Verify the published version through the current retrieval path and retain enough provenance to explain which source and version were returned.

Can Web Scraper Cloud build the complete RAG data pipeline?

Web Scraper Cloud can provide sitemap-based collection, JavaScript-capable execution, scheduled or API-controlled jobs, webhooks, parsing, data-quality controls and data delivery. A downstream service remains responsible for document identity, cross-run change detection, versioning, chunking, embeddings, index publication and application retrieval policy.

Build for verified freshness

A useful RAG data pipeline does not merely collect web pages and place embeddings somewhere nearby. It maintains a verifiable relationship between a live source, a versioned document and the chunks currently eligible for retrieval.

Design that relationship first. Use stable identities, preserve provenance, distinguish observation from change, update incrementally, handle removals conservatively and block suspicious runs from reaching production. The result is not merely a fresher knowledge base. It is one the team can explain when the answer matters.