The Myth That Public Data Is Free to Scrape

The most repeated line in scraping is also the most misleading: if the data is public, it is free to scrape. It is not that simple, and treating it as a rule gets people into trouble. Public visibility is one factor among several. What actually determines your risk is how you access the data, what kind of data it is, what terms you agreed to, and what you do with it afterward. Public is a starting point for the analysis, not the end of it.

Why is public data not automatically free to scrape?

Because access and content are separate questions. Data can be public in the sense that anyone can view it in a browser, and still be protected by terms of service, by database rights in some jurisdictions, by privacy law if it concerns people, and by anti-circumvention rules if you had to defeat something to reach it. Visibility does not erase any of that.

The clearest example is personal data. A person's profile might be publicly viewable, but privacy regimes like GDPR still treat scraping and processing that data as regulated activity. Public visibility gives you no free pass on personal data, which is exactly why I keep scraping personal data and compliance as a separate discipline from ordinary scraping. The rules are different and stricter.

What actually determines my scraping risk?

Four things, roughly in order of weight.

How you access it. Scraping data you can reach as an anonymous public visitor is very different from scraping data behind a login you had to create and agree to terms for. The moment you accept terms of service, you have a contract, and those terms usually restrict automated collection. I cover why the access method matters so much in is web scraping legal.

What the data is. Factual business data sits on much safer ground than creative content protected by copyright, or personal data protected by privacy law. A price is a fact. A person's photo is not.

Whether you circumvented anything. If you defeated a technical barrier to get in, you have moved into far riskier territory. Reading an open page is not the same as breaking a wall.

What you do with it. Collecting data for internal analysis is different from republishing it, and republishing a substantial copy of someone's database can trip database and copyright protections even when each individual fact is public.

Does robots.txt make scraping legal or illegal?

Neither, and this is its own myth. Robots.txt is a request, not a law. Ignoring it does not automatically make you a criminal, and following it does not automatically make you clean. It matters as evidence of intent and as a norm, but it is not the binding line people treat it as. I unpack that specifically in does robots.txt bind your scraper. Respect it as a default courtesy, but do not mistake it for the whole legal question.

How should I actually operate given all this?

Stop looking for a single rule that makes everything safe, because there is not one. Instead, reduce risk on every axis you control. Collect factual, non-personal data where you can. Access it as a public visitor without defeating barriers. Respect the norms including robots.txt and reasonable rate limits, which also keeps you from getting blocked, per web scraping mistakes that get you blocked. Use the data internally rather than republishing wholesale. And when the stakes are real, get actual legal advice for your jurisdiction and use case. I am an operator, not your lawyer.

The reason this matters for how you build: a defensible scraping operation is one you can explain. Clean access patterns, respect for norms, clear lineage on what you collected and why. That is easier to run on infrastructure designed for it than on a tangle of scripts nobody can audit. Owning your collection stack, which I argue for in own your scraping infrastructure, also means owning the record of how you operated. PyroSync is built around that kind of controlled, auditable collection, because in scraping, being able to show how you worked is part of staying safe.