Job board scraping: Build a reliable job dataset

Job board scraping becomes difficult when the output must remain complete, current and explainable across recurring runs. A scraper can finish successfully and still return duplicate listings, stale vacancies, missing detail pages or records that cannot be traced back to what the source showed.

The reliable approach is to design the dataset before automating collection. Give every source listing an identity, preserve its history, normalise fields without discarding the original values, and define how a posting becomes active, updated or inactive. This guide covers permitted public job boards, company career pages and accessible applicant tracking system pages.

Define the dataset contract before building the scraper

Start with the decision the dataset needs to support. A recruitment lead feed, salary benchmark, labour-market dashboard and job-search product do not require the same coverage, history or freshness.

Write a short operational contract covering:

• Scope: sources, countries, job families, languages and page types included.
• Freshness: how soon a new or changed posting must appear downstream.
• Record grain: what one row represents and how multi-location jobs are handled.
• Required fields: the minimum values that make a record usable.
• History and delivery: current state, append-only observations or versioned upserts.
• Failure policy: which gaps are tolerated and which stop publication.
• Ownership: who reviews failed checks and source changes.

“Collect all software jobs every day” is not a data contract. A usable version defines the collection scope, delivery deadline, required identity fields and the record-count or field-completion failures that quarantine a run.

This makes reliability measurable. The broader production web scraping checklist can then be used as a release gate for scheduling, recovery, delivery and ownership.

Treat a source listing and a vacancy as different entities

The most important modelling decision is to separate two ideas:

1. A source listing is one advertisement observed on one source. It has a source-specific ID or URL, raw fields and observation history.
2. A vacancy cluster links source listings that probably refer to the same underlying opening across an employer site, ATS page and one or more job boards.

These are not interchangeable. One vacancy can appear through several channels, while two jobs can share a title and employer but differ by location, seniority, requisition or team.

Use the source listing as the auditable base record. Deduplicate repeat observations of that listing deterministically. If the business needs a market-level view, build a separate vacancy-clustering layer and retain the evidence behind each match. A fuzzy rule should never silently delete source records.

Create a source registry

A recurring project needs an inventory of its sources, not just a list of start URLs. For each source, record:

• source name, type and approved entry URLs;
• searches, categories, employers and regions covered;
• listing and detail page templates;
• pagination method and tested stopping condition;
• JavaScript or interaction requirements;
• identifiers and lifecycle signals available;
• collection schedule and expected record range;
• owner and date of the last successful manual verification.

This registry defines coverage. If a source suddenly returns no jobs, you can distinguish a real empty result from an uncollected category, broken selector or failed pagination path.

Choose sources according to the dataset rather than assuming every job board adds equal value. Company career pages may be closest to the current application state. Accessible ATS pages often use repeatable templates across employers. Job boards provide broader discovery and categorisation, but can introduce reposts, delayed updates and duplicates.

Prefer an authorised feed or API when it provides the required fields, history, freshness and usage rights. Use a visual sitemap when the data is available on permitted public pages and the team needs direct control over navigation and extraction. Choose custom code when the collection logic or deployment requirements genuinely require it.

Design a schema that preserves evidence

Keep three layers in the dataset:

• Provenance: where, when and in which run the observation was collected.
• Raw values: what the page visibly displayed or embedded.
• Normalised values: consistent fields used for search, joins and analysis.

A practical starting schema is:

Group	Recommended fields	Why they matter
Source identity	source_name, source_job_id, source_url, canonical_url	Supports traceability and deterministic upserts
Role	title_raw, title_normalised, description_raw, employment_type	Preserves source evidence while enabling comparison
Employer	company_raw, company_normalised, company_domain	Supports grouping without overwriting the displayed employer
Location	location_raw, country, region, city, remote_type, applicant_region	Separates workplace location from remote eligibility
Compensation	salary_raw, salary_min, salary_max, currency, salary_period	Prevents a number from losing its unit or currency
Timing	date_posted_raw, date_posted, valid_through, first_seen_at, last_seen_at, last_changed_at	Separates source dates from your observations
State	listing_state, state_reason, missing_run_count	Makes active and inactive status explainable
Audit	scrape_run_id, captured_at, page_type, content_hash, validation_status	Supports debugging, replay and change detection

The Schema.org JobPosting vocabulary is a useful checklist for dates, hiring organisation, employment type, salary and location. It does not prove that every source supplies those fields or keeps them accurate.

When JSON-LD or other structured data is available, extract it alongside the visible values. Compare the two rather than trusting whichever is easiest to parse. A conflicting salary, location or expiry date should create a validation flag. The broader guide to what data can be extracted from a website explains why embedded data and the rendered page can represent different states.

Build extraction around listing and detail pages

Most job sources have at least two relevant page types:

1. Listing or search pages discover detail URLs and may provide summary fields.
2. Detail pages contain the full description, application state, requisition ID, salary, location details and structured data.

Capture listing-page context, but do not assume it is the complete record. Cards may contain shortened titles, partial locations, relative dates and no stable ID. A label such as “3 days ago” also needs the collection time if it will be converted into a date.

In Web Scraper, define one repeated job-card wrapper and place the title, employer, location and link selectors beneath it. The sitemap and selector-tree documentation explains how record boundaries prevent a title from one card being paired with the location from another.

Follow each unique detail link and test more than the easiest posting. A representative fixture set should include:

• the first and a later pagination page;
• remote, hybrid, on-site and multi-location jobs;
• postings with and without salary data;
• salary ranges and single-value salaries;
• expired, removed and legitimately empty pages;
• alternative templates used by different employers; and
• content revealed by clicking, scrolling or JavaScript.

Test relationships as well as individual selectors. A preview can contain every expected value while still combining fields from neighbouring cards.

Use the simplest driver that repeatedly reaches the required state. Fast extracts returned HTML without executing JavaScript. FullJS is appropriate when job cards, pagination or detail content require rendering or interaction. Neither choice guarantees compatibility with every site.

Before automation, preview and validate the sitemap, run a limited collection and inspect the actual exported records.

Create stable source keys and normalise carefully

Within one source, prefer identity in this order:

1. a stable source-provided job or requisition ID;
2. a stable canonical detail URL;
3. a documented source-specific composite key; or
4. a fallback fingerprint built from stable raw fields.

Never discard the original URL. Remove tracking parameters only when they are known not to distinguish postings.

Add a content hash built from the selected fields that matter to the dataset. The source key answers which listing was observed. The hash answers whether its relevant content changed.

This supports idempotent updates:

• the same key and hash refreshes last_seen_at;
• the same key with a changed hash creates a new version;
• a new key inserts a source listing; and
• an absent key adds lifecycle evidence rather than immediately deleting the record.

Normalisation should improve comparison without erasing meaning. Preserve Remote, UK only while separating remote status from applicant geography. Keep £50k-£65k per annum beside parsed bounds, currency and period. If a page says “posted three days ago”, store the calculated date as an inferred value tied to captured_at, not as an exact source date.

Unknown values should remain unknown. Do not assign a country to “Remote - Europe”, convert an unqualified salary into an annual amount or fill a missing salary with an estimate presented as source data.

Deduplicate in two passes

Pass 1: deterministic source deduplication

Use the source key to prevent the same listing from being inserted repeatedly. Check exact source IDs and canonical URLs within each source. This pass should be conservative and explainable.

Pass 2: cross-source vacancy clustering

Create a separate match candidate using several signals:

• employer identity or company domain;
• requisition ID or application URL;
• normalised title and seniority;
• location and remote restrictions;
• posting-date window; and
• description similarity, department and employment type.

No single field is enough. The same company may have several “Software engineer” vacancies, while one opening may use different titles across channels.

Use confidence bands. High-confidence matches can share a vacancy cluster. Ambiguous matches should remain separate or enter review. Preserve every contributing source listing and the match evidence so a bad merge can be reversed.

Model lifecycle and freshness explicitly

A job dataset needs states, not only rows.

State	Evidence	Default action
new	First valid observation of the source key	Insert and record first_seen_at
active	Seen again and still open	Refresh last_seen_at
updated	Same key with a changed relevant-content hash	Save a new version and the changed fields
missing_pending	Not found in an otherwise valid run	Add absence evidence without declaring expiry
inactive	Explicit closure, past validThrough, removed detail page or enough confirmed valid misses	Close the active interval and record the reason
reopened	A credible listing returns after inactivity	Start a new active interval and preserve history

Google’s job-posting guidance recognises several publisher-side expiry signals, including a past validThrough, removal of JobPosting markup and a detail page returning 404 or 410. These are useful evidence, but a missing search result is not proof of closure. Ranking, filters, pagination changes and failed collection can all hide an active listing.

Choose refresh frequency from the maximum delay the use case can tolerate. Then separate recurring work into four paths:

• Discovery sweep: revisit listing pages to find new source keys.
• Detail refresh: recheck new or changed postings.
• Missing-item verification: revisit listings that disappeared from a valid discovery run.
• Periodic reconciliation: perform a broader crawl to repair missed changes and confirm state.

Measure accepted-data freshness, not only schedule frequency. Useful metrics include time from observation to accepted delivery, age of the last valid source run, detail-page coverage, records outside their expected refresh window, and new, updated, missing and inactive counts by source.

Put quality gates between scraping and publication

Separate three outcomes:

1. Execution completed: the scraper stopped running.
2. Extraction succeeded: expected pages and fields produced records.
3. Dataset accepted: the run met the contract and may update trusted state.

A page can return 200 OK but no usable data, such as a consent screen, access challenge, JavaScript shell or repeated first page.

Apply checks at three levels:

• Page checks: expected page type, final URL and positive landmarks; recognised empty or removed states; no challenge, login or error markers; later pages do not repeat page one.
• Record checks: required identity and provenance fields; valid date ordering; salary units and currencies; structured-data conflicts; duplicate and null rates.
• Run checks: expected source and pagination coverage; plausible record-count change; required-field completion; acceptable failed, empty and no-value page rates.

Set thresholds from each source’s successful history. One large board can hide the complete failure of a smaller source when only global totals are checked.

Quarantine a run that fails the contract and retain the previous accepted snapshot. Web Scraper Cloud data-quality controls can monitor record counts, page outcomes and field completion. Cross-source matching, historical comparison and lifecycle policy remain downstream responsibilities.

Preserve observations and history

Do not overwrite the only copy of the latest accepted record. Maintain:

1. raw observations tied to a collection run;
2. a current source-listing view for normal queries; and
3. version history or events for created, changed, missing, inactive and reopened states.

Store the source, sitemap and schema version, run ID, collection time, validation result and downstream import state. Reprocessing the same run should not create duplicate versions or repeat lifecycle events.

This structure lets operators determine whether a change came from the labour market, the source website, the sitemap or a transformation rule.

Keep collection within a reviewed scope

Public visibility is not a complete permission analysis. Review source terms, robots rules, rate limits, authentication boundaries, relevant privacy and intellectual-property requirements, and the intended use of the data.

Collect only the fields needed for the defined purpose. Job pages can expose individual recruiter details, but company-level hiring analysis may not require them. Define retention, access and correction policies when personal data is involved.

Web Scraper is a strong fit for tested, accessible public job boards, company career pages and ATS pages. It is not the default choice for social platforms such as LinkedIn or for large collections behind login.

Automate a tested sitemap with Web Scraper Cloud

Use the free Web Scraper browser extension to build and test the sitemap on representative pages:

1. Define listing record boundaries and follow detail links.
2. Extract source identity, raw fields and lifecycle signals.
3. Test pagination, layout variants, missing values and closed postings.
4. Run a limited local scrape and inspect the dataset.
5. Import the tested sitemap into Web Scraper Cloud.
6. Configure the driver, schedule, parsing, data-quality thresholds and delivery.
7. Send accepted output to the downstream system for normalisation, vacancy clustering and lifecycle updates.

Cloud can schedule recurring runs or launch an existing sitemap through the API. A completion event can then trigger downstream processing. The choice between webhooks and polling affects how completion is detected, not whether the dataset is correct.

Prove identity, lifecycle and acceptance on one source before adding more. More sources multiply ambiguity until those rules are stable.

Frequently asked questions

What is the best unique key for a job posting?

Use a stable source-provided job or requisition ID where possible. A canonical detail URL is the next useful option. There is no universal cross-source key, so probable duplicates should be clustered from several signals rather than merged by title alone.

Should a missing job be marked as closed immediately?

No. Absence from one valid run is evidence, not proof. Mark the listing as pending and look for an explicit closed state, expired validThrough, removed detail page or repeated absence according to a source-specific policy.

Can Web Scraper deduplicate jobs from different websites?

Web Scraper collects, parses, monitors and delivers structured records from tested sitemaps. Cross-source vacancy matching and historical lifecycle logic should be handled in the downstream database or data pipeline so the match evidence and corrections are preserved.

Build a public job-listing dataset with the visual sitemap builder, validate it against representative vacancies, and move it to Web Scraper Cloud when the workflow needs recurring runs, monitoring and automated delivery.