Production web scraping checklist before deployment

A prototype proves that a scraper can collect the right data under supervision. Production readiness means proving that it can run unattended, reject bad output, deliver an agreed dataset and recover without creating duplicates or silently publishing errors.

Use this production web scraping checklist as a release gate. If the scraper has no measurable acceptance criteria, named owner or tested recovery path, it is not ready to feed a business system, even if the latest manual run looked correct.

Start with an operational contract

Write down what the scraper promises before choosing a scheduler. “Scrape the catalogue every day” is not a publication standard.

For a product-monitoring workflow, the contract might be:

Define seven parts:

Contract part	Question to settle
Scope	Which sites, page types, categories, regions and exclusions belong in the run?
Freshness	When must collection start, finish and become available to consumers?
Correctness	Which fields, relationships and source-page states make a record acceptable?
Failure policy	Which gaps are tolerated, quarantined or severe enough to stop publication?
Delivery	What file, table or API receives an accepted dataset, and is delivery a snapshot, append or upsert?
Access	Have target rules, terms, data types, rights and intended use received appropriate review?
Ownership	Who receives an alert, decides whether to publish and communicates with consumers?

This is deliberately narrower than capacity planning. If your main problem is worker count, throughput, partitioning or a durable URL frontier, use the guide to scaling web scraping from thousands to millions of pages.

1. Build representative acceptance fixtures

Build a small, versioned fixture set covering source variations that could change the output or execution path.

Fixture	What it tests	Example assertion
Normal listing and detail page	Base extraction and record grouping	Every item produces one record with the correct key
Alternative template	Layout variation	The same fields retain the same meaning
Discounted or variant item	Conditional values	Regular and discounted prices do not swap columns
Missing optional field	Null handling	A missing value stays null rather than shifting another value
Later pagination page	Navigation coverage	Records continue beyond the first page
JavaScript or interaction-dependent page	Page readiness	Extraction begins only after the required state appears
Required regional or language version	Geographic and locale state	Currency, language and product identity match the requested context
Legitimate empty or removed page	Terminal-state handling	The outcome is recorded without inventing a record
Consent, login or challenge page	Wrong-page detection	The response is rejected even if it returns 200 OK

For each fixture, record its source, expected page class, positive and forbidden markers, record range, required values and allowed nulls. Use saved artefacts to make extraction tests reproducible and live cases to test navigation, JavaScript, location and current access conditions.

Web Scraper's guide to preview and validate a sitemap covers item variations, later pagination, alternate layouts and interaction-loaded content. Inspect the resulting records and a limited export, not only highlighted elements. Supported fixtures must pass and unknown states must fail closed.

2. Treat versions, configuration and secrets as release inputs

A production run should be reproducible from named inputs. Keep a release record that connects:

• the scraper or sitemap version;
• schema and transformation versions;
• start URLs and scope rules;
• driver, page-readiness, location and session settings;
• request interval, timeout and schedule time zone;
• destination, owner and release date;
• the last known-good rollback version.

Separate development and production configuration. A local test must not write to a production table, activate its schedule or send trial data to a customer.

Store credentials outside source code, sitemap names and exports. Scope access, plan rotation and keep secrets out of logs, screenshots and page artefacts. The OWASP Secrets Management Cheat Sheet provides vendor-neutral guidance for CI/CD systems, containers and secret stores.

3. Define scheduling and execution behaviour

A cron expression is not a scheduling policy. Define late-start, overlap, missed-run, time-zone and dependency behaviour before enabling automation.

Confirm the time zone and daylight-saving policy, overlap rule, missed-run and duplicate-safe backfill behaviour. Manual and automated launches should use the same production configuration, validation and publication gates.

Choose the trigger that matches the business process:

Trigger	Best fit	Production question
Time-based schedule	Data is needed at predictable intervals	What happens if the previous run is still active?
API trigger	Another workflow decides when data is needed	How is the request correlated with the resulting dataset?
Manual launch	Controlled exception or recovery	Who is authorised, and how is the run documented?

Use the simplest execution path that produces the required page. In Web Scraper Cloud, Fast extracts returned HTML without executing JavaScript, while FullJS supports JavaScript and interactions. Scrolling, clicking and Website State Setup require FullJS. Test the production choice against the fixtures.

The Cloud scheduler uses a selected time zone, waits when the previous scheduled job for the same sitemap is still running and preserves the sitemap version captured by an already queued job. Test these behaviours before launch.

4. Classify outcomes before adding retries

“Retry on error” is too broad. Some failures are transient; others prove that the request, page state, selector or destination needs attention.

Observed outcome	Default production action
Connection reset, transient DNS problem or selected 5xx response	Retry with a capped delay and attempt budget, then quarantine
429 Too Many Requests	Honour a usable Retry-After, reduce target pressure and retry only within the budget
401, persistent 403, login page or challenge	Pause and diagnose access, permission, session or configuration
Expected 404, removed listing or recognised empty result	Record the terminal business state according to the data contract
Unexpected redirect, 404 or regional page	Quarantine as a page-classification failure
200 OK with missing records, consent or challenge content	Fail page validation and inspect the returned artefact
Selector or schema validation failure	Preserve evidence and repair the affected template before rerunning it
Destination timeout after an uncertain write	Reconcile by run and idempotency key before sending again

HTTP defines 429 as rate limiting, while Retry-After can specify a date or delay. Rising latency, connection drops and challenge pages can also justify reducing pressure.

A successful status code is not a data-quality result. The guide to 200 OK but no data explains why final routes, page markers, rendered state and extracted fields need independent checks.

Give each class an attempt or time limit and an end state. Quarantine exhausted work; never relabel an unexplained failure as legitimate emptiness.

5. Treat the dataset as an interface

Treat output consumed by another system as an interface. Specify:

• what one row or object represents;
• stable entity and observation keys;
• field names, types and required status;
• the meaning of null, empty text, zero and “not available”;
• source URL, capture time, target region and other provenance;
• currencies, units, time zones and normalisation rules;
• how variants, removals, unavailable items and corrections are represented;
• schema version and compatibility policy;
• snapshot, append or update semantics at the destination.

For JSON delivery, JSON Schema can express machine-checkable constraints. Structural validity is only one layer. A price can be a valid decimal and still belong to the wrong currency, seller or product.

6. Put validation between completion and publication

A process can finish successfully while returning incomplete or plausible-looking wrong data. Make publication a separate decision with five levels of checks:

1. Coverage: Did the run process the intended sites, categories, page types and known inventory?
2. Page correctness: Did each URL return the expected route, template, language, currency and page state?
3. Record validity: Are keys present, types valid, relationships consistent and required fields populated?
4. Dataset plausibility: Are record counts, uniqueness, field-fill rates and changes from the previous accepted run within explained bounds?
5. Delivery readiness: Does the final schema match the consumer contract, and are all required parts accepted?

Derive thresholds from representative history and business tolerance. Give optional fields separate rules so expected blanks do not hide failures. Prefer positive assertions, such as confirming the requested product ID and template marker, rather than checking only for the absence of “captcha”.

Do not publish directly over the last trusted dataset. Write a versioned candidate, validate it and then promote it. If validation fails, keep the previous accepted version available.

Web Scraper Cloud's data-quality controls can check record count, failed and empty page percentages and field population. A data-quality failure does not change a completed scraping job into a failed job, so downstream automation must not equate “finished” with “accepted”.

7. Make every run explain itself

Observability should let an operator answer three questions:

1. What did we intend to collect?
2. What pages and data did we actually receive?
3. What dataset, if any, did we publish?

Give every run a stable ID and manifest containing its scope, versions, configuration, timing, outcome counts, validation result and delivery status. Carry the same ID into transformation and the destination.

Monitor operation and data together:

View	Useful signals	Question answered
Execution	Start delay, duration, state, retries and stalled work	Did the run operate as expected?
Source response	Status and final-route mix, page states, driver and location	Did the source return the intended pages?
Extracted data	Records by page type, field fill, duplicates and schema failures	Does the candidate satisfy the data contract?
Freshness	Oldest required observation and missed windows	Is the business still using current data?
Delivery	Validation decision, import status, rejected rows and acknowledgement	Did the accepted dataset reach its consumer?

Dashboards support investigation; alerts must prompt action. Send an urgent alert when a critical dataset is late, validation blocks publication or delivery is uncertain. Google SRE's monitoring guidance recommends reserving alerts for urgent, actionable conditions.

8. Assign ownership and write an outcome-first runbook

Name the scraper owner, data owner, delivery owner, business approver and backup responder. One person may hold several roles, but each responsibility must be explicit.

Write the runbook around what the operator observes:

• No job started: Check the schedule, credentials and deployment state.
• Job is still running: Check progress, target delays and the next collection window.
• Pages failed or are empty: Inspect URLs, failure reasons and captured artefacts before changing the driver, route or selector.
• Record count is normal but fields are missing: Compare templates, parent selectors, schema and field-population trends.
• Validation failed: Stop publication, record the failing rule and isolate the affected scope.
• Delivery is uncertain: Query the destination by run ID before retrying the write.
• Bad data was published: Disable further publication, identify affected consumers, restore or mark the last accepted dataset and reconcile downstream writes.

Include dashboard links, safe queries, contacts and approved artefact locations. Test that the backup responder can use it without private context from the original developer.

9. Stage the launch and keep delivery reversible

Increase the consequence of each test in controlled stages:

Stage	What runs	Promotion gate
Production rehearsal	Production configuration on the fixture set, with no business delivery	Versions, secrets, logs and validation behave as expected
Shadow run	Live representative scope, with output quarantined	Results match source samples and the current accepted process
Delivery canary	One target, category or destination partition	Import is safe to repeat, traceable by run ID and reversible
Limited schedule	Several recurring cycles at bounded scope	Freshness, alerts, ownership and repair work are acceptable
Full schedule	Agreed production scope	Every hard gate passes and the recovery drill succeeds

Define stop conditions before launch, such as an unknown page state, missing category, required-field collapse or uncertain write. Operators must be able to stop collection and publication independently.

Keep the last accepted dataset and prior scraper version. Reverting extraction does not undo data already consumed by another system.

10. Run a recovery drill before go-live

Test failures across different boundaries:

• interrupt a run and recover only the intended scope;
• deliver the same completion event twice without creating duplicate imports;
• make the destination unavailable after extraction finishes;
• fail a required-field check while total record count remains normal;
• return a consent or challenge page with 200 OK;
• deploy a broken selector, pin the previous version and recover affected data;
• rotate a production credential and confirm that logs do not expose it.

Rollback can require three separate actions:

1. Restore the previous scraper and configuration.
2. Keep or restore the previous accepted dataset.
3. Pause, reverse or reconcile any downstream import made from the bad candidate.

Record detection and recovery time, manual steps and dependence on the original developer. Convert private knowledge into a runbook action or automated check, then repeat the drill.

Web Scraper Cloud webhooks can trigger downstream work when a job reaches a final status, but delivery attempts may repeat. Test an idempotency key based on the job and status, and keep large imports outside the webhook request.

11. Review the first production cycles

Compare the first complete cycles with the contract. Check freshness, missed source variations, alert usefulness and reliance on private knowledge. Confirm that published records remain traceable to a run, version, source and validation result.

Update fixtures, thresholds, ownership and the runbook from this evidence.

The final production web scraping checklist

Treat these as release gates.

Contract and evidence

• Scope, freshness, correctness, failure and delivery expectations are written.
• Access rules, target terms, data rights, personal data and intended use have received appropriate review.
• Representative live and saved fixtures pass, while unknown states fail closed.
• Source-matched samples confirm business meaning, not only schema validity.

Deployment and execution

• Scraper, schema, transformation and configuration versions are traceable.
• Production configuration and secrets are separated from development.
• The production execution settings and scheduling behaviour match the accepted test.
• Retryable, terminal and repair-required outcomes have separate policies and limits.

Data and delivery

• The data contract defines keys, types, nulls, provenance, schema changes and write semantics.
• Coverage, page, record, dataset and delivery checks run before publication.
• A finished job can be blocked from publication when validation fails.
• Writes are traceable and repeatable, and the last accepted dataset is retained.

Operations and recovery

• Run manifests connect intended scope, execution, validation and delivery.
• Alerts lead a named owner or backup to a documented action.
• Collection and publication can be stopped independently.
• Canary delivery and the three-part recovery path have been exercised.
• Duplicate events, interrupted runs, bad selectors and destination failures have been tested.

Make the release decision

Decision	When it applies	Required action
Deploy	Every hard gate passes, the canary and recovery drill succeed, and owners approve	Enable the agreed schedule and review the first production cycles
Hold	A fixable control or evidence gap remains	Keep unattended publication disabled and assign an owner and resolution date
Stop	Access or intended use is unacceptable, no accountable owner exists, or bad output cannot be contained and recovered	Redesign or cancel the production workflow

Deploy a tested sitemap without operating the execution layer

Build and validate the sitemap in the browser extension, then import or synchronise it to Web Scraper Cloud. Cloud provides remote execution, scheduling, monitoring, retries, notifications and automated delivery.

Before scheduling, run a limited production-configured job, inspect its records and page outcomes, set quality thresholds and test the destination. Automatic data export supports configured destinations; APIs and webhooks support custom workflows.

Cloud manages the execution environment. Your team still owns the data contract, acceptance thresholds, downstream writes and publication decision. Successful execution does not prove that the dataset is correct.

Promote the sitemap only after its canary delivery and recovery drill succeed.