How Often Should You Recrawl a Site?
How often you recrawl a site should match how fast the data moves and how fast users need it, not a default. Here is how to set crawl frequency without wasting money.
There is no universal recrawl frequency, and anyone who gives you one number is guessing. How often you should recrawl a site is set by two things: how fast the underlying data actually changes, and how fresh your users need it to be. Crawl faster than that and you burn money refetching identical pages. Crawl slower and you serve stale data someone is making decisions on. The right cadence is the slowest one that still meets the user's freshness requirement, and it is different for every field on the page, not just every site.
What decides how often I should recrawl?
Two independent questions, and you need both answers before you pick a number.
First, how fast does the data change? A stock price changes by the second. A product price changes daily or when a sale runs. A company's address changes maybe once in years. A directory listing might be static for months. You are wasting every fetch that returns data identical to last time, so the change rate sets your floor.
Second, how fresh do users need it? This is a product question, not a technical one. A pricing intelligence client trading on your data needs it within minutes. A monthly market report can run on data that is a week old. Freshness is a feature you are choosing to deliver, and it should be set by the decision your data feeds, which is the same reasoning I apply to crawl scheduling in how to schedule and orchestrate scraping jobs.
Your recrawl frequency is the tighter of those two constraints. There is no point recrawling hourly if the data changes weekly, and no point recrawling weekly if your users need daily.
Should every page be recrawled at the same rate?
No, and treating them the same is where most of the waste is. Within a single site, different pages change at wildly different rates. A retailer's homepage and category pages update constantly. A product's specs page might never change while its price changes daily. Crawling all of them on one schedule means either over-crawling the stable pages or under-crawling the volatile ones.
Segment your targets by observed change rate. Put the fast-moving pages on a tight cadence and let the stable ones ride a slow one. You learn the change rate by watching it: track how often each page's content hash actually changes over a few weeks, then set each page's cadence to match. Pages that never change get demoted. Pages that change every crawl get promoted. This adaptive scheduling is how you stop paying premium frequency for stable data.
How do I avoid paying for pages that did not change?
This is where recrawl frequency and incremental crawling meet. Even on a tight schedule, you should not be refetching and reparsing pages that are unchanged. Use conditional requests and change detection so a "recrawl" of a stable page costs almost nothing. A server that honors If-Modified-Since returns 304 with no body, so you confirm freshness for the price of a header exchange. I lay out the full mechanics in incremental crawl vs full recrawl.
Done right, you can run a high nominal frequency cheaply, because most of those runs cost almost nothing when the data has not moved. The expensive fetches only happen when there is actually something new to get. That decouples "how often I check" from "how much I pay," which is the key to affordable freshness. And since every avoided fetch is proxy and compute you did not spend, this feeds straight into the unit math in how to calculate the true cost per page of scraping.
What is a sane starting point?
When you have no data yet, start conservative and let observation tune it. Daily is a reasonable default for most commercial data. Run it for a couple of weeks with change detection on, measure how often each page actually changed, then adjust: promote the volatile pages, demote the static ones, and set the overall cadence to the freshness your users truly need rather than the freshness that feels impressive.
Resist the urge to crawl often because you can. Frequency is not freshness if the data did not change, it is just cost. The discipline is to match cadence to real change rate and real user need, segment your pages, and lean on incremental fetching so checking often stays cheap. Set it that way and you deliver fresh data without paying for stale refetches. If you want a crawler that learns each page's change rate and schedules recrawls adaptively for you, that is part of what PyroSync handles.