Resume interrupted scraping jobs without duplicates

To resume interrupted scraping jobs safely, do not continue from a worker's last loop index. Reconstruct unfinished work from durable task state, assume that a task may run more than once and make repeated writes harmless with stable keys and atomic checkpoints.

This guide focuses on recovery mechanics: task states, leases, transaction boundaries, retries and targeted reruns. For broader decisions about capacity, concurrency and URL frontiers, see scaling web scraping from thousands to millions of pages.

Resume from durable work, not worker memory

A log entry such as last page: 842 is not a reliable checkpoint. It becomes ambiguous when pages run concurrently, retries finish out of order or one page writes data before the process stops.

Start by defining one independently repeatable task. For product monitoring, that might be one canonical product URL for one daily collection window. Give each responsibility its own identifier:

Identifier	What it represents	Example basis
run_id	One intended collection and candidate dataset	Dataset plus collection window
task_key	One logical unit of work, stable across retries	Dataset, canonical URL, page type and collection window
attempt_id	One execution of a task	A new UUID for every attempt
entity_key	The real product, variant, offer, job or listing	Stable source or business identifier
observation_key	One entity state in one collection window	Entity key plus collection window

A practical task key is:

dataset_id + canonical_url + collection_window + page_type

Do not include the attempt number. A retry must collide with the original logical task rather than appear as new work. Store the scraper and schema versions as attributes so the history still shows which version produced the result.

URL identity is not business identity. A current-state table may upsert by product ID, while a price-history table needs a product ID plus collection date. Event data may use a source event ID. A database can enforce a key, but it cannot decide whether a product, variant and seller offer are separate entities.

Separate execution, acceptance and delivery

One success flag cannot describe the complete pipeline. Track three related decisions:

• Task execution: pending → leased → committed, with branches to retry_wait or quarantined.
• Dataset acceptance: collecting → candidate → accepted → published, or candidate → rejected.
• Delivery: unsent → sending → delivered, with failed or uncertain writes requiring reconciliation.

A task can finish while returning a consent page or invalid values. A dataset can be accepted while its destination import fails. In that case, retry delivery rather than collecting every page again.

This separation also makes the checkpoint meaningful. committed should mean that the accepted output and task state are durable. It should not mean only that a request returned.

Store the minimum recovery schema

A relational database can implement the core guarantees with five records:

Record	Minimum purpose and fields
runs	Scope, collection window, scraper/schema version, candidate state, acceptance state and publication state
tasks	Task key, run, partition, URL, state, attempt count, next eligible time, lease owner, lease expiry, lease token and last error class
attempts	Attempt ID, task key, recovery batch, worker, timing, response status, outcome and diagnostic artefact reference
observations	Observation key, entity key, collection window, payload hash, source task key, validation state and payload
outbox	Event ID, run ID, event type, payload, creation time and delivery time

Enforce task_key and observation_key with primary or unique constraints. Ten retries may create ten attempt records for diagnosis, but they should still produce one logical accepted observation.

The outbox closes a separate failure gap. Accepting a dataset in the database and notifying another system are two operations. A transactional outbox stores the publication event in the same transaction as the acceptance change, then sends it separately and retryably.

Claim tasks with expiring leases

A worker should lease a task, not permanently remove it when processing begins. If the worker disappears, the lease expires and the task becomes eligible again.

Managed queues commonly provide this through a visibility timeout. Amazon SQS, for example, temporarily hides a received message but can deliver it again if it is not deleted. Its standard queues use at-least-once delivery, so consumers must tolerate duplicate messages.

A database-backed queue can claim work in a short transaction:

BEGIN;

WITH claim AS (
  SELECT task_key
  FROM scrape_tasks
  WHERE next_attempt_at <= now()
    AND (
      state IN ('pending', 'retry_wait')
      OR (state = 'leased' AND lease_until < now())
    )
  ORDER BY priority DESC, next_attempt_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE scrape_tasks AS task
SET state = 'leased',
    lease_owner = :worker_id,
    lease_until = now() + interval '5 minutes',
    lease_token = lease_token + 1,
    attempt_count = attempt_count + 1
FROM claim
WHERE task.task_key = claim.task_key
RETURNING task.*;

COMMIT;

Choose the lease duration from measured task times. Extend it with a heartbeat when browser work can exceed the initial lease. Every heartbeat and final commit should match the current lease_token. If an old worker resumes after another worker has claimed the expired task, the token prevents the stale worker from overwriting the newer result.

Make the result commit the checkpoint

A checkpoint is a committed fact, not a progress message. After retrieval and validation, save the attempt, observation and task state together:

BEGIN;

INSERT INTO scrape_attempts
  (attempt_id, task_key, outcome, http_status, payload_hash)
VALUES
  (:attempt_id, :task_key, 'accepted', :http_status, :payload_hash);

INSERT INTO observations
  (observation_key, entity_key, collection_window,
   payload, payload_hash, source_task_key)
VALUES
  (:observation_key, :entity_key, :collection_window,
   :payload, :payload_hash, :task_key)
ON CONFLICT (observation_key)
DO UPDATE SET
  payload = EXCLUDED.payload,
  payload_hash = EXCLUDED.payload_hash,
  source_task_key = EXCLUDED.source_task_key;

UPDATE scrape_tasks
SET state = 'committed',
    lease_owner = NULL,
    lease_until = NULL
WHERE task_key = :task_key
  AND lease_owner = :worker_id
  AND lease_token = :lease_token;

-- Roll back unless exactly one current lease was committed.

COMMIT;
-- Acknowledge the queue item only after COMMIT succeeds.

If the task update matches no current lease, roll back the whole transaction. Do not keep an observation written by a worker that had already lost ownership.

The final ordering is critical: commit first, acknowledge second. If the commit succeeds but acknowledgement is lost, the queue may deliver the task again. The stable key and upsert make the repeat harmless. If acknowledgement happens first and the commit fails, the work can disappear without a stored result.

This does not guarantee exactly-once execution. It creates one logical stored outcome from repeatable, at-least-once work.

Test the crash windows with a complete example

Each interruption point should produce a defined recovery action:

Failure point	Required result
After lease, before retrieval	The lease expires and another worker can claim the task
After retrieval, before commit	No durable success exists, so the task runs again
During the transaction	Attempt, observation and checkpoint roll back together
After commit, before acknowledgement	Redelivery finds the logical result and adds no duplicate
After acceptance, before notification	The outbox retains the unsent event
After an uncertain destination response	The importer checks the stable key before repeating the write

Suppose a daily price collection contains 100,000 product tasks. The process stops with 63,700 accepted observations, 35,900 pending tasks and 400 tasks whose leases have not yet expired.

Keep the 63,700 accepted observations. When the 400 leases expire, make those tasks eligible again alongside the pending work. Reuse their original task and observation keys, but create new attempt IDs.

Assume 280 of the uncertain tasks had committed observations before the interruption. Their repeated attempts encounter existing observation keys and do not append 280 extra rows. The remaining 120 complete normally. Coverage is calculated from logical tasks or accepted observations, never from attempt count.

Now suppose one retailer's selector produced invalid prices for 4,000 pages. Preserve the original manifest, quarantine that retailer partition, repair and test the selector, then create a recovery batch under the original run for those task keys. Use new attempt IDs and the original observation identities. Revalidate the repaired partition and then the complete candidate dataset before publication.

If the incorrect values were not published, the repaired rows can replace the rejected candidate values. If consumers already received them, issue a versioned correction or superseding observation so the audit history remains intact.

Retry by failure class

Retry only when another attempt may succeed without a code, configuration or permission change.

Outcome	Default action
Connection reset, transient DNS failure or selected timeout	Retry with capped exponential backoff and jitter
Transient 500, 502, 503 or 504	Retry within an attempt and elapsed-time budget
429 Too Many Requests	Honour a usable Retry-After, reduce pressure and retry later
401, persistent 403, login or challenge page	Quarantine and diagnose access or session conditions
Confirmed 404, 410 or removed listing	Record the terminal source state
200 OK with a consent page, challenge or wrong template	Reject or quarantine the page state
Selector, schema or required-field failure	Quarantine the affected template or partition until repaired
Uncertain destination write	Query by the idempotency or run key before retrying
Retry budget exhausted	Move the task to a reproducible dead-letter or quarantine path

RFC 6585 defines 429 and allows a Retry-After header. RFC 9110 permits that value to be an HTTP date or a number of seconds, so handle both. For other transient errors, one full-jitter implementation calculates:

maximum_delay = min(cap, base × 2^attempt)
delay = random(0, maximum_delay)

AWS's analysis of exponential backoff and jitter explains why randomisation reduces clustered retry traffic.

HTTP success is not data success. A scraper can receive 200 OK with a login screen, soft block or empty application shell. The guide to diagnosing 200 OK responses with no data covers that classification in detail.

Match the checkpoint to discovery

The correct checkpoint depends on how the project finds work:

• Fixed URL inventory: Persist every logical task and recover unfinished keys.
• Cursor pagination: Save the next cursor only in the transaction that accepts the current page.
• Dynamic link discovery: Persist both pending URLs and the seen set before acknowledging the parent task.
• Changing offset pagination: Treat the offset as navigation state, not proof of entity coverage. Store records by stable entity keys and validate the final inventory.

Do not advance the accepted checkpoint after merely downloading bytes. The page must pass the expected-page and extraction checks. Rejected output should remain recoverable.

Rerun only the affected scope

Use a targeted rerun rather than restarting a valid collection:

1. Freeze the original run manifest and retain the last accepted dataset.
2. Select tasks by an explicit partition, page type, error class or failed validation rule.
3. Repair the selector, page-state logic or destination and record the new version.
4. Create a recovery batch under the original run for those logical task keys.
5. Execute with new attempt IDs while retaining the intended observation keys.
6. Upsert the recovered observations rather than appending them blindly.
7. Recalculate validation for the repaired partition and affected whole-dataset checks.
8. Promote the candidate only after acceptance passes, retaining recovery provenance.

The recovery batch records another attempt to produce the same intended observations. It is not a new collection window.

Apply the pattern with Web Scraper Cloud

Web Scraper Cloud manages remote sitemap execution and automatically re-scrapes empty and failed pages. If pages remain, inspect their failure reasons and available artefacts, fix persistent access or sitemap problems and use Continue where appropriate.

Your integration still owns entity identity, idempotent database writes, dataset acceptance and publication. A practical workflow is:

1. Create an internal run ID and pass it as custom_id when launching a Cloud job.
2. Receive the final-status webhook and return 2xx promptly.
3. Queue a small “synchronise this scraping job” task instead of importing a large dataset inside the webhook request.
4. Download the current data with scrapingjob_id and upsert records under stable observation keys.
5. Evaluate coverage, page outcomes, required fields and the data-quality result before publication.

Cloud webhook notifications may be delivered more than once. Continue can also produce another valid notification for the same job. Do not permanently discard every later event merely because scrapingjob_id + status appeared before. Make synchronisation repeatable so a later notification can import the updated result safely.

Data quality control is a separate boundary: a quality failure does not turn a successfully completed scraping job into a failed job. Publication logic must check both decisions.

When a repair can be represented as a smaller URL inventory, one-time custom start URLs can narrow a Cloud launch without permanently changing the sitemap. This is a scoped rerun option, not customer control over Cloud's internal queue or checkpoints.

For file delivery, automatic data export sends the complete dataset after a job finishes. A manual download made while the job is running is only a partial snapshot and should not be promoted as the recovered complete dataset.

Cloud does not expose customer-controlled per-page leases, checkpoint transactions or its exact retry backoff. Those are custom-pipeline patterns around the managed execution layer.

Prove recovery before depending on it

Run a controlled drill that interrupts work before commit and again after commit but before acknowledgement. Expire a lease while a slow worker is active, deliver the same webhook twice, run Continue, return 429 with both forms of Retry-After and make a destination time out after an uncertain write.

Verify that:

• no intended task is lost;
• repeated execution creates no duplicate logical observation;
• every attempt remains traceable;
• quarantined output cannot be published;
• a repaired partition leaves accepted partitions unchanged;
• delivery can repeat without triggering another collection; and
• the previous accepted dataset remains available until its replacement passes.

The broader production web scraping checklist covers ownership, alerts, deployment and rollback beyond this recovery protocol.

Recover execution without compromising the dataset

Web Scraper Cloud can manage remote execution, automatic page retries, inspection and export. Your pipeline can then add stable observation keys, acceptance gates and repeat-safe delivery at the systems it controls.

Build and validate the sitemap first, then define the recovery contract before the first large recurring run. Review how Cloud scraping jobs are configured and monitored.