Webhooks vs polling for scraping job results

Webhooks usually beat polling for detecting when an asynchronous scraping job has finished: they reduce unnecessary API calls and notify your pipeline with less delay. Polling is still the simpler option for low-volume jobs, outbound-only networks or systems that cannot receive public HTTPS requests.

For recurring commercial datasets, the strongest design is often hybrid. Use a webhook as the fast completion trigger, process the result idempotently outside the request, and poll only to reconcile jobs whose notification may have been missed. Whichever pattern you choose, treat job completion and dataset acceptance as separate decisions.

Webhooks vs polling: choose a failure model, not just a transport

An asynchronous scraping API returns before the crawler has finished. Your application therefore needs a second mechanism to learn whether the job reached finished, failed or another terminal state.

Polling and webhooks solve that notification problem in opposite directions:

• With polling, your application asks for the current job status on a schedule.
• With a webhook, the scraping service sends an HTTP request to your endpoint when the job reaches a relevant state.

The usual summary, “webhooks are efficient and polling is simple”, is true but incomplete. A production decision also depends on what happens when requests time out, notifications are repeated, events arrive late, workers restart or a job finishes with poor data.

Decision factor	Polling	Webhook	Hybrid
Detection delay	Controlled by poll interval	Usually low after the provider emits the event	Usually low, with delayed recovery if an event is missed
Requests while a job runs	Repeated status reads	None from the consumer	Occasional reconciliation reads
Network requirement	Outbound API access only	Public HTTPS receiver	Public receiver plus outbound API access
Primary failure risk	Poller stops, exceeds a deadline or hits rate limits	Endpoint outage or finite delivery attempts leave a missed event	More moving parts, but each path covers the other
Best fit	Low-volume or delay-tolerant work	Simple event-driven workflows with tolerable manual recovery	Important recurring data pipelines

A webhook does not make a scrape run faster. It reduces the interval between the provider reaching a terminal state and your system noticing. Likewise, polling is not automatically reliable merely because the client initiates it. A poller that loses its local job list or stops running can miss completions too.

What reliable webhook handling actually requires

Webhook delivery crosses a network boundary. A lost response can cause a retry after your system accepted the first request, while an endpoint outage longer than the sender’s retry window can leave an event undelivered. Design for duplicates, possible gaps and late events rather than assuming one notification always creates one downstream action.

Acknowledge after durable acceptance, before expensive work

A safe webhook path is deliberately short:

1. Accept the HTTPS request and parse the documented content type.
2. Validate required fields and allow only known event or status values.
3. Correlate the provider job with a persisted internal workflow.
4. Record the receipt and enqueue or otherwise durably schedule processing.
5. Return a successful response.
6. Let a worker retrieve, validate and import the dataset.

The acknowledgement belongs after durable acceptance, because acknowledging before the event is stored creates a loss window. It belongs before the download and import, because holding the request open makes a timeout and duplicate delivery more likely. If your system cannot persist or enqueue the event, return a failure response and let the sender apply its documented retry policy.

Idempotency is about effects, not merely detecting repeats

Deduplicating the HTTP request is useful, but the more important goal is to make downstream effects safe to repeat. An idempotent importer can run again without creating duplicate rows, publishing the same dataset twice or triggering the same alert repeatedly.

Useful controls include:

• a unique import-generation record for each accepted dataset snapshot;
• upserts based on stable business keys, such as product ID plus observation time;
• an atomic transition from pending to processing to committed;
• a dataset fingerprint and record counts stored beside the import.

A provider job ID is valuable for correlation, but do not assume it is always a complete event ID. Some systems can issue more than one legitimate event during a job’s lifecycle. The safest design makes the materialisation step idempotent even when notification-level deduplication is imperfect.

Treat a webhook as a prompt to reconcile, not permission to overwrite local state blindly. A worker can fetch the current job, apply only valid transitions and keep execution, import and data acceptance as separate state dimensions.

Design a poller from latency and load

Polling replaces inbound infrastructure with a scheduling problem. It is often the right trade when there are few jobs, the acceptable delay is generous or security policy makes a public callback endpoint impractical.

Start with the completion-detection service-level objective rather than copying a universal interval. Let T be a fixed polling interval. If completion can occur uniformly between polls, the added detection delay ranges from almost zero to just under T, with an average of roughly T / 2. For a job lasting D seconds, the poller makes approximately ceil(D / T) status requests, depending on when the first check occurs.

For an illustrative 40-minute job:

Poll interval	Approximate average detection delay	Approximate status reads
60 seconds	30 seconds	40
15 seconds	7.5 seconds	160

Reducing the interval by four improves average detection delay by four, but also creates roughly four times as many status requests. At fleet scale, that trade can affect both your scheduler and the provider’s rate limit.

Use a bounded adaptive schedule

A fixed interval is easy to reason about, but an adaptive schedule is often kinder to long-running crawls:

1. Wait until the minimum plausible runtime has passed.
2. Poll at a moderate interval during the expected completion window.
3. Increase the interval gradually while a healthy job remains non-terminal.
4. Cap the interval so worst-case detection delay remains acceptable.
5. Stop at a local deadline and mark the job unknown or stale for investigation.

Add jitter so a batch of jobs does not poll simultaneously. AWS and Google Cloud document backoff and jitter for retry bursts. Keep this separate from healthy polling: “still running” follows the polling schedule, while network errors, retryable 5xx and rate-limit responses use bounded error backoff.

On HTTP 429, follow the server’s returned timing rather than retrying immediately. Web Scraper’s API specification documents X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset on this response, so clients should honour the returned reset information rather than hard-code example quota values.

Run polling as a central scheduler rather than one permanent loop per job. Persist a next_check_at value, claim due jobs in controlled batches and enforce a request budget. This keeps concurrency and API traffic visible when many jobs start together. Poll the known job resource, not a broad paginated job list, and stop normal polling at a documented terminal state or local deadline.

Why a hybrid is stronger for recurring scraping

For price monitoring, listings, stock data, leads or property feeds, silence is dangerous. A missed completion can leave yesterday’s dataset in production without raising an obvious error. Constant polling avoids dependence on a callback but wastes reads during normal operation. Webhook-only processing is efficient but cannot recover automatically after every possible endpoint outage.

A hybrid design uses two paths:

• Fast path: the webhook is durably accepted and feeds the normal import worker.
• Recovery path: a scheduled reconciler finds locally tracked jobs that have remained non-terminal or unprocessed beyond their expected window, reads their current status and sends discovered terminal jobs through the same worker.

The reconciler should be low frequency compared with a normal poller. Its purpose is to repair stale state, not compete with the webhook. Select its threshold from expected job duration and business freshness requirements. A daily catalogue feed may tolerate a longer recovery window than an intraday stock monitor.

Both paths must converge on one idempotent processing function. If webhook handling and reconciliation run separate import code, their behaviour will drift and they can race each other. A single operation key or atomic state transition should ensure that one worker commits while another observes the completed outcome.

Make reconciliation observable as well. Track how many jobs it recovers, how long they stayed stale and why the fast path failed. A hybrid with an unmonitored scheduler only moves the hidden failure somewhere else.

Keep one job ledger for both paths. At minimum, separate the state needed to answer where the workflow stopped:

Field	What it tells you
Execution state	Whether the remote job is queued, running or terminal
Import state	Whether results are pending, processing, committed or failed
Acceptance state	Whether the dataset is unchecked, accepted, quarantined or rejected
Webhook receipt time	Whether the completion signal arrived and how late it was
next_check_at	When reconciliation may poll the job again
Dataset version or fingerprint	Whether repeated processing produced the same snapshot

Whether completion is discovered by webhook or polling, both paths should update this ledger and call the same idempotent worker.

Three practical choices

• Daily competitor price monitoring: Use a webhook to start retrieval as soon as the scrape finishes. Reconcile jobs that remain unresolved near the reporting deadline, then validate product IDs, price fields and record-count changes before updating the reporting table.
• API-triggered marketplace collection: Pass an internal batch reference through custom_id, persist the returned scraping job ID and use the webhook to queue an idempotent upsert. A repeated notification should not duplicate listings or publish the same snapshot twice.
• Restricted corporate network: If policy prevents a public inbound endpoint, use a central poller with controlled concurrency, jitter, an explicit deadline and alerts for overdue jobs. This is a valid polling-first design, but request volume must be sized for peak concurrent jobs.

Finished does not mean the dataset is correct

Completion detection answers “has execution stopped?” It does not answer “is this dataset suitable for downstream use?” A scrape can reach a terminal state after receiving access-denied pages, empty templates, consent screens or a changed site layout. It can also produce records whose critical fields are blank or whose count is far outside the expected range.

Add an acceptance gate after download and before publication. For recurring datasets, useful checks include:

• expected record-count range and change from the previous run;
• uniqueness and stability of business keys;
• fill rate for required columns;
• failed, empty and no-value page proportions;
• schema and type checks;
• domain rules, such as non-negative prices or valid listing URLs;
• sample comparison against the rendered source when the site has changed.

Quarantine questionable output instead of replacing a known-good dataset. Preserve job metrics, quality results, the imported record count and a fingerprint so operators can tell whether a retry produced a genuinely new snapshot.

Web Scraper makes this distinction explicit. A configured data-quality check can fail without changing a successfully completed scraping job into a failed job. Its job monitoring separates records, failed pages, empty pages and no-value pages, while data-quality controls expose thresholds and an overall result. If transport succeeded but the returned page was wrong, the guide to diagnosing a 200 response with no usable data provides the next diagnostic step.

Implement the pattern with Web Scraper Cloud

Web Scraper Cloud automates an existing sitemap that has been built and tested. Its API is not an arbitrary URL-in, dataset-out endpoint. Once the sitemap is production-ready, the following workflow keeps completion detection and data acceptance separate.

1. Start and correlate the job

Create the job through POST /scraping-job. Supply a caller-managed custom_id that maps to an internal batch or workflow, and persist the returned scrapingjob_id before considering the start step complete. custom_id is a correlation field, not a documented idempotency guarantee, so your application remains responsible for its uniqueness and lifecycle.

The short knowledge-base guide shows how to start jobs and retrieve results through the API; the architecture here adds durable state, recovery and quality control around that sequence.

2. Receive the documented payload

According to the current Web Scraper webhook documentation, Cloud sends a form-encoded HTTP POST, not a JSON body. It contains scrapingjob_id, status, sitemap_id, sitemap_name and, when supplied, custom_id. Notifications are sent when a job reaches finished, stopped or failed. The dataset is not embedded in the webhook.

Use a public HTTPS endpoint, validate required fields and reject unknown statuses. The documentation suggests a secret token in the webhook URL as one possible origin check. It does not document an HMAC header, delivery ID, timestamp or source-IP range, so do not assume those features exist.

3. Meet the acknowledgement deadline

The endpoint must return 2xx within 10 seconds. Web Scraper documents a retry when the endpoint times out or returns 300 or higher, with the first retry after five seconds and the second after ten seconds. Those are the documented attempts, so do not assume an unlimited retry window.

Validate, durably enqueue and acknowledge. Keep dataset download, parsing and database writes out of the request. If the first response is lost or late, a retry may repeat the same notification, so the handler and worker must be safe to run more than once.

4. Handle Continue without discarding real work

Web Scraper can send another valid notification for the same scraping job after failed or empty URLs are processed again with Continue and the job returns to a final status. Because the documented payload has no delivery ID or continuation generation, (scrapingjob_id, status) cannot always distinguish a delivery retry from a legitimate later same-status completion.

Use that pair to correlate or coalesce immediate processing, but do not make a permanent “seen once, ignore forever” decision from it. Re-read the current job, materialise results idempotently and compare an import generation, dataset fingerprint or relevant metrics. This recommendation is an inference from the documented payload and Continue behaviour, not a vendor-provided exactly-once guarantee.

5. Fetch state, data and quality separately

The Web Scraper Cloud API exposes a single-job status resource, data downloads and a data-quality result. Use GET /scraping-job/{scrapingJobId} for known-job reconciliation. Download results through GET /scraping-job/{scrapingJobId}/{extension}, where the documented formats are json, csv and xlsx; JSON output is newline-delimited JSON, which can be streamed instead of loaded fully into memory.

After downloading, query GET /scraping-job/{scrapingJobId}/data-quality and apply your own business checks before promoting the snapshot. A stopped or failed job normally belongs in an operational failure path. A finished job proceeds to validation, but only an accepted dataset should replace the current downstream version.

6. Reconcile overdue jobs

Run a scheduled query over jobs whose expected completion window has expired. Poll Web Scraper’s single-job endpoint and route any terminal result through the same worker used by webhooks. Add the provider identifiers, processing attempts, record and page metrics, quality outcome and downstream commit to the shared job ledger. This turns “the data did not update” from a mystery into a traceable sequence.

Choose the simplest design that survives your real failures

Use polling alone when jobs are infrequent, a small detection delay is acceptable, outbound API access is easy and running a public endpoint plus queue would be disproportionate. Use a webhook-only path when the workflow can tolerate manual recovery from a missed notification.

For recurring datasets that drive commercial decisions, choose the hybrid. It preserves the webhook’s low latency and low idle request volume, while reconciliation repairs jobs left stale by a callback outage. In all three designs, the non-negotiable controls are persisted job IDs, bounded retries, idempotent effects, an explicit stop condition and a dataset-quality gate.

Web Scraper Cloud provides the final-state webhook, status and download APIs, job metrics and separate quality results needed for this pattern. The reliable pipeline is completed by the state machine around those capabilities: accept durably, process asynchronously, reconcile stale jobs and publish only validated data.

Build and test the sitemap in the browser extension, then start a 7-day free trial to test the webhook, reconciliation and validation flow in Web Scraper Cloud before connecting it to production data.