LLM Parsing vs CSS Selectors for Scraping: When to Use Each

The question I get most from people modernizing a scraper is whether they should throw an LLM at parsing and stop writing selectors. The honest answer: use selectors where the structure is stable, use an LLM where it is not, and never use an LLM where a selector would do, because it costs orders of magnitude more per page. This is not a religious choice between old and new. It is an economic one, made per target, and getting it wrong in either direction either wastes money or wastes engineering time. I run both in the same pipeline and route each site to the cheaper method that works.

Why not just use an LLM for everything?

Because cost and speed. A CSS selector or XPath extraction is nearly free: microseconds of CPU, no network call, deterministic output. Running a language model on the page is a different universe of cost. Each page is an inference call with real latency and real per-token pricing, and at scale that can become the single largest line in your budget, dwarfing proxies and even headless rendering. I break down where these costs land in how to calculate the true cost per page of scraping, and model inference is the newest way to accidentally make a crawl unprofitable.

There is also the accuracy trap. People assume the LLM is more reliable because it is smarter. It is not more reliable in the way that matters: it is non-deterministic and it will confidently return a plausible wrong value, the same failure mode as a broken selector but harder to detect because the output looks reasonable. Whichever method you use, you have to validate the output, which I cover in how to validate scraped data quality before it poisons you. The LLM does not remove the need for validation; it makes it more important.

When are CSS selectors the right call?

Whenever the page structure is stable and consistent, which is most high-volume targets. If you are scraping a single site's product pages by the million, they share a template, and one well-written set of selectors extracts every page for a fraction of a cent. Writing those selectors is a one-time cost amortized across millions of pages. An LLM would pay full inference on every one of those pages to extract data a selector already handles perfectly.

Selectors win on: high volume, consistent templates, structured metadata already in the page like JSON-LD, and any case where you can find the underlying JSON API. The stability concern people raise, that selectors break on redesigns, is real but manageable with the anchoring and drift-detection techniques in how to handle website layout changes that break scrapers. A selector that breaks loudly and is cheap to fix beats a model that costs a hundred times more on every single page to avoid a repair you do a few times a year.

When is an LLM actually worth it?

When the structure is chaotic and the volume is low. The LLM earns its cost exactly where selectors fail: pages with no consistent template, wildly variable layouts, unstructured free text you need to interpret, or a long tail of sites where writing bespoke selectors for each one costs more engineering than the data is worth.

Concrete cases where I reach for a model:

• The long tail. Scraping ten thousand different sites once each, where no two share a structure and per-site selectors are uneconomical.
• Unstructured extraction. Pulling structured fields out of prose, like parsing a job description paragraph into salary, seniority, and requirements.
• Low volume, high variability. A few thousand pages where inference cost is trivial and selector engineering is not.

In those cases the model's flexibility is worth its price because the alternative, hand-writing parsers for endless variation, costs more in engineering than the inference does in compute.

How do I decide per target?

Run the numbers on volume times per-page cost, both ways. For a high-volume consistent target, selectors are cheaper by orders of magnitude and the decision is easy. For a low-volume chaotic target, the LLM's inference cost is trivial and it saves you real engineering, so it wins. The crossover is where volume is high enough that inference cost adds up but structure is variable enough that selectors are painful, and there you often use a hybrid: a model to figure out the extraction pattern once, then generated selectors to run it cheaply at scale.

The meta-point is that this is a routing decision inside one pipeline, not a platform you pick. Owning that routing, so each target goes to the cheapest method that returns correct data, is exactly the kind of control I keep in house rather than renting, which is the argument in own your scraping infrastructure. Selectors for the stable high-volume core, an LLM for the messy long tail, validation on both, and the cost math deciding the line between them. If you want a scraping layer that already routes each target to the right parsing method for you, that is what PyroSync is built to do.