How to Calculate the True Cost Per Page of Scraping

The true cost per page of scraping is not your proxy bill divided by page count. That number is a lie people tell themselves right before a project stops being profitable. Real cost per page is proxy bandwidth plus compute plus retries plus parsing plus storage plus the human time to fix what breaks. Until you can write that number on one line, you do not know whether your scraping is making money or quietly burning it. I compute it for every crawl before I commit, because at a million pages a half cent error is five thousand dollars.

What actually goes into cost per page?

Six things, and most people count one.

• Proxy or bandwidth cost. Residential proxies are billed per gigabyte and cost far more than datacenter IPs. A page that pulls in images and trackers you do not need is paying residential rates for junk.
• Compute. An HTTP fetch is nearly free. A headless browser render is not. This single choice can swing cost per page by ten to fifty times, which is why I decide it deliberately in when to render JavaScript for scraping.
• Retries. A page that fails and succeeds on the third attempt cost you three fetches, not one. If your success rate is 70 percent, your real per page cost is roughly 1.4 times the naive number before anything else.
• Parsing. CPU to turn HTML into fields. Cheap with selectors, expensive if you run a model on every page.
• Storage and transfer. Raw HTML, parsed records, and the egress to move them. Small per page, real at scale.
• Maintenance. The site changes, the scraper breaks, and someone fixes it. Amortize that engineer time across the pages the fix keeps running.

Why is the retry rate the number that hurts most?

Because it multiplies everything above it. Every retry pays proxy cost again, compute again, and bandwidth again. A crawl running at a 50 percent first attempt success rate is not a little more expensive. It is double, and often worse once you add the pages that never succeed and get abandoned after the retry cap.

This is why block rate is a cost metric, not just a reliability metric. Cutting your block rate from 30 percent to 5 percent can cut cost per page by a third on its own. Most of that fight is proxy strategy and request shaping, which I go deep on in the hard part of scraping is proxy management and web scraping mistakes that get you blocked. Fix the block rate first. It is usually the cheapest lever with the biggest effect on the unit number.

How do I actually measure it?

Run a real sample, not a spreadsheet guess. Crawl ten thousand pages of the target with your actual stack. Then divide real totals by pages successfully extracted, not pages attempted. That denominator matters. A page you fetched but could not parse into usable data cost money and returned nothing, so it belongs in the numerator and not the denominator.

Instrument four counters: requests sent, pages successfully parsed, bytes transferred, and browser seconds consumed. From those plus your provider rates you can build the whole per page number. Track it per target site, because a JavaScript heavy site behind aggressive anti bot defenses can cost twenty times what a static site costs, and blending them hides which crawls are underwater.

What is a good cost per page?

There is no universal number, but there are ranges. A static HTML page over datacenter proxies with clean parsing should land in fractions of a cent. A JavaScript rendered page over residential proxies with a moderate block rate can run several cents. If you are paying more than a few cents per static page, something is wrong: you are rendering when you do not need to, retrying too much, or paying residential rates for datacenter appropriate targets.

The reason I keep this stack in house rather than renting per successful request is control over exactly these levers. A managed API charges you a flat rate per page whether that page cost them a tenth of a cent or five cents to fetch, and they price for the expensive case. Owning the pipeline lets me route each target to the cheapest path that works, which is the argument I make in full in own your scraping infrastructure. Get the six inputs on one line, watch the retry rate, and price every crawl before you scale it. If you would rather have a provider that already optimizes these levers per target, that is what PyroSync is built for.