Incremental Crawl vs Full Recrawl: How to Decide
An incremental crawl only fetches what changed since last time. A full recrawl refetches everything. Here is how to choose and cut most sites' cost by 90 percent.
If you refetch every page on every run, you are paying full price to rediscover data you already have. An incremental crawl fetches only what changed since last time. A full recrawl fetches everything. For most real datasets the answer is incremental, and switching can cut a crawl's cost by ninety percent because on any given day the vast majority of pages did not change. The whole game is knowing which pages did change without fetching all of them to find out.
What is the difference between incremental and full recrawl?
A full recrawl treats every run as if it were the first. It rediscovers URLs, fetches each one, and reparses everything. It is simple, it is correct, and it is expensive. You use it when the dataset is small, when you cannot detect change cheaply, or when you suspect your last crawl was wrong and want a clean baseline.
An incremental crawl carries state between runs. It remembers what it saw last time and spends its budget only on what is new or changed. New URLs get fetched. Changed pages get refetched. Everything else gets skipped. The cost drops toward the change rate of the site instead of the size of the site.
The trap is that incremental is not free to build. You need a reliable way to detect change and somewhere to store crawl state. If you get change detection wrong, you skip pages that actually changed and your data goes stale without any error firing. That silent staleness is the real risk, and it is worse than an obvious failure because nothing alerts you.
How do I detect what changed without fetching everything?
This is the entire problem, and there is a ladder of methods from cheapest to most expensive.
- Sitemaps with lastmod. Many sites publish an XML sitemap with a last modified date per URL. Read it, compare against your last crawl, fetch only the newer ones. Nearly free when it exists and is honest, which it often is not.
- HTTP conditional requests. Send If-Modified-Since or If-None-Match with the ETag you stored. A well behaved server returns 304 Not Modified with no body, so you pay almost nothing to confirm a page is unchanged.
- Listing pages and feeds. For catalogs and news, the index page or RSS feed tells you what is new without crawling every detail page. Crawl the shallow index, diff against known IDs, fetch only the new detail pages.
- Content hashing. When none of the above is trustworthy, fetch the page but hash the meaningful content. If the hash matches your stored one, skip the parse and downstream write. You still paid the fetch, but you saved parsing, dedup, and storage churn.
Use the cheapest method the site supports. The first two mean you barely fetch anything to know what to fetch. That is where the ninety percent saving comes from.
When should I still do a full recrawl?
Incremental drifts. Small errors in change detection accumulate, and over months your dataset develops blind spots: pages that changed in a way your detector missed, URLs that quietly dropped off, records that should have been deleted but linger. A periodic full recrawl is the reconciliation that catches this drift.
I run incremental daily and a full recrawl on a slower cadence, weekly or monthly depending on the source. The full pass is the ground truth that corrects whatever the incremental passes missed. Think of incremental as the fast path and full recrawl as the audit. You need both. This scheduling logic is part of the same orchestration problem I cover in how to schedule and orchestrate scraping jobs.
How does this connect to freshness and cost?
Freshness is a product decision, not a technical default. A pricing dataset that clients trade on needs hourly incremental crawls. A directory that changes monthly does not. Match your crawl frequency to how fast the data actually moves and how fast your users need it, which is the question I work through in how often should you recrawl a site.
The reason incremental matters so much is unit economics. Every page you skip is proxy cost, compute, and bandwidth you did not spend, and those are the exact inputs to the number I break down in how to calculate the true cost per page of scraping. Full recrawl is the expensive default that feels safe. Incremental is the discipline that makes large scraping affordable. Build the change detection, keep a periodic full pass as your safety net, and let the site's real change rate set your bill. If you want incremental crawling and 304 handling done for you out of the box, that is part of what PyroSync provides.