How to Deduplicate Scraped Data Without Losing Records

Deduplicating scraped data is not running DISTINCT on a table. That deletes real records and keeps garbage. The actual problem is deciding what makes two records the same thing when the source gave you no stable ID, formatted the same product three different ways, and listed it on four pages. Get the identity key right and dedup is easy. Get it wrong and you either drown in duplicates or silently delete data you paid to collect. I treat the dedup key as a design decision made before the first crawl, not a cleanup step bolted on after.

What makes two scraped records the same?

You need an identity key: the minimal set of fields that uniquely names a thing in the real world. For a product it might be brand plus model plus a normalized title. For a company it might be domain plus normalized name. For a job posting it might be employer plus title plus location plus posted date. The key is never "everything," because everything includes the fields that legitimately differ across captures, like price and timestamp.

Pick the key from what stays stable across pages and across time. If a field changes when you recrawl the same item, it does not belong in the identity key. If two genuinely different items can share a field, that field alone is not enough. This sounds obvious and it is the step almost everyone skips, then spends weeks patching downstream.

How do I handle exact duplicates versus near-duplicates?

Two different problems, two different tools.

• Exact duplicates come from crawling the same URL twice or the item appearing on a listing page and a detail page. Kill these with a content hash. Normalize the record, hash the identity fields, and reject on collision. This is cheap and catches most of the volume.
• Near-duplicates are the hard ones. "Sony WH-1000XM5" and "Sony WH1000XM5 Wireless Headphones" are the same product with different strings. Exact hashing misses them completely. You need normalization first (lowercase, strip punctuation, collapse whitespace, standardize units) and then fuzzy matching or a blocking key that groups candidates before you compare them.

Do the cheap normalization before you reach for fuzzy matching. Most "near duplicates" collapse into exact duplicates once you standardize casing, whitespace, and unit formatting. Fuzzy matching every record against every other record is quadratic and will not scale, so you block first: group by a coarse key like brand, then only compare within the group.

Where in the pipeline should dedup happen?

At write time, not query time. If you dedup only when you read, every consumer re-solves the same problem and they will not agree. Enforce identity at the point data enters your store, with a unique constraint on the normalized key so the database refuses duplicates instead of trusting your code to remember. This is part of why storage layout and dedup are the same conversation, which I get into in how to store scraped data at scale.

Enforcing at write time also forces the merge decision up front. When a new record collides with an existing one, you have to decide: keep newest, keep most complete, or merge fields. That decision belongs in your pipeline design, next to validation, which is why I treat dedup as one stage of the flow in how to design a data pipeline for scraped data rather than a separate cleanup job.

What should I do when records conflict?

When two records claim to be the same thing but disagree on a field, you need a merge rule, not a coin flip. The rules I use, in order of usefulness:

• Newest wins for volatile fields like price and stock. The most recent crawl is the truth.
• Most complete wins for descriptive fields. A record with a filled brand beats one with a null brand.
• Source priority when you scrape multiple sites. Trust the authoritative source over an aggregator.
• Keep both with provenance when you genuinely cannot decide. Store which crawl each value came from so a human can adjudicate later.

The thread through all of this is that dedup is a data quality decision, not a database trick. Bad dedup poisons everything downstream, and you often will not notice until a customer does. That is the same reason I validate hard on the way in, which I cover in how to validate scraped data quality. Define the identity key first, normalize before you match, enforce at write time, and write your merge rules down. Do that and dedup stops being a recurring fire. If you would rather have the identity, normalization, and merge logic handled for you, that is built into PyroSync.