Run a Cloud scraping job

Configure the sitemap run

Open the sitemap in Web Scraper Cloud and use the Scrape tab to configure how the next job should run. There is no separate job-creation step.

The available run settings include:

  • Request interval
  • Page load delay
  • Driver
  • Proxy
  • Priority
  • Scrape with custom start URLs

After configuring the sitemap, select Test scrape to validate the configuration or Scrape to start the full job.

Request interval and page load delay

Request interval

Request interval controls the minimum interval between page requests made by the scraping job. Increasing the interval reduces the request rate. Reducing it can make the job complete faster, provided the target website continues to respond correctly.

Page load delay

Page load delay controls how long Web Scraper waits for the page before data extraction begins. Some websites need additional time for content to render or for JavaScript-generated values to become available.

Both settings have a minimum value of 500 ms. The default values are the recommended starting point. If faster scraping is required, lower values can be tested. Use a Test scrape after changing them and confirm that the expected data is still returned before using the lower values for a full job.

The two settings control different parts of execution. A lower request interval increases how quickly new page requests can be made, while a lower page load delay allows extraction to begin sooner after a page is loaded. Reducing either value too far can result in incomplete or missing data if the target website cannot respond or render quickly enough.

Use custom start URLs for a one-time job

A sitemap already contains its configured start URL, so the normal scraping workflow does not require reviewing or re-entering start URLs.

Enable Scrape with custom start URLs when the sitemap structure should remain the same but a single job needs to start from different URLs.

For example, a sitemap may be configured for one product category, but the same selector structure also works for another category. Custom start URLs allow that second category to be scraped for one job without changing the sitemap itself.

Custom start URLs override the sitemap start URL for that job only. They do not permanently modify the sitemap. To change the start URL for future jobs, update the sitemap itself.

Custom start URLs can be supplied as text or from a file. When entered as text, place each URL on a new line. Custom start URLs are processed in jobs of up to 20,000 URLs. A launch can contain up to 200,000 URLs; Cloud automatically splits a 200,000-URL launch into 10 scraping jobs.

Set job priority

Priority controls the order in which waiting jobs are selected when a parallel task becomes available. It does not interrupt or change the priority of jobs that are already running.

The available priority levels are:

  • Highest
  • High
  • Default
  • Low
  • Lowest

For example, if all three available parallel tasks are already running and five jobs with Default priority are waiting, a newly launched High-priority job is placed ahead of those Default-priority jobs. It becomes the next eligible job to start when a parallel task becomes available.

Use higher priority for jobs that should start ahead of other queued work. Changing priority does not provide additional parallel tasks and does not stop an existing job to make room for the higher-priority job.

Validate with Test scrape

Select Test scrape before committing to a full scraping job when validating a new sitemap or configuration.

A Test scrape processes up to 500 links and does not consume URL credits. The 500-link limit is based on links processed, not on records returned. A link counts towards the limit even when it returns no data.

Use the test result to check selector output, page behavior, driver and proxy configuration, and any changes made to request interval or page load delay.

Start the full job

When the configuration has been validated, select Scrape to start the full scraping job. The job either starts immediately when a parallel task is available or enters the queue according to its priority.

Monitor the job

The scraping jobs table shows the job status, scraped pages, failed pages, empty pages, no-value pages, record count, duration, proxy, data quality result, driver and creation time.

Use Inspect and Preview to review the result before downloading or automating further runs.

Related