Web Scraping and Automated Data Collection: Access, Contract, and Copyright

Whether an automated collector is lawful is not one question. It is four. Does the collection break a computer-access law? Does it breach a contract the collector agreed to? Does it burden someone else's machines enough to be a tort? Does it copy protected expression? Those four theories sit in different legal layers, are brought by different parties, and turn on different facts. A project can be clean on one and badly exposed on another, which is why "the data was public" answers far less than people assume.

Four theories, not one verdict

Most scraping disputes are argued on some combination of the same four claims. Sorting them apart first is the only way to see where a particular project is actually thin.

The split between layers matters. Computer-access law, copyright, and the anti-circumvention rules of the Digital Millennium Copyright Act are federal. Contract, trespass to chattels, and most computer-crime and consumer-privacy statutes are state law, which means the answer can change with the forum even on identical facts.

Why the access statute stopped being the main event

The federal Computer Fraud and Abuse Act, codified at 18 U.S.C. § 1030, punishes accessing a computer without authorization or in excess of authorized access. For years site operators treated it as a general-purpose anti-scraping weapon. Two developments cut that back.

In the hiQ Labs v. LinkedIn litigation, the Ninth Circuit held that scraping profile information that anyone could view without logging in did not violate the statute's access provisions. The reasoning turned on the absence of an authentication gate: if no permission was ever required to see the page, there was no authorization to exceed.

The Supreme Court narrowed the statute further in Van Buren v. United States (2021), reading "exceeds authorized access" as a gates-up-or-down question — whether the person was entitled to obtain that information at all, not whether they used it for a forbidden purpose. Misusing information you were allowed to retrieve is a different problem from breaking in.

Note: Neither ruling makes scraping lawful. They make one federal theory harder to win on public pages. Everything behind a login, a paywall, an API key, or a technical barrier sits in different territory, and circumventing a technical protection measure can raise a separate DMCA question.

The state layer that took over: contract and server burden

As the federal access theory contracted, the practical fight moved to contract. If a collector created an account, accepted terms during signup, or used credentials issued under an agreement, the site's terms of service become a promise the collector made — and automated collection is prohibited in a large share of them.

Enforceability is its own question, and it depends on how the terms were presented rather than on what they say. A clearly presented agreement that required an affirmative click is treated very differently from small print linked in a page footer, a distinction covered in our piece on when an online agreement actually binds.

Trespass to chattels is the second state-law lever. It is an old tort about interfering with someone else's property, applied to servers. Courts generally want evidence of real impairment — degraded performance, bandwidth costs, capacity consumed — not merely an unwanted visitor. A polite crawler that respects rate limits is a poor target; an aggressive one that knocks a site over is a good one.

Watch out: State computer-crime statutes are not carbon copies of the federal one. Some reach conduct the federal statute no longer covers, and a few states attach their own consequences to bypassing access controls. Assuming a national rule because of a federal appellate decision is the single most common error here.

Personal information adds another layer entirely. Collecting and reselling consumer records can pull a project into state privacy regimes and, in several states, into the registration and deletion duties described in our guide to data broker registration and deletion requests. The FTC also publishes business guidance on unfair and deceptive data practices, which reaches how collected data is later used and described.

Copyright: facts are free, the packaging often is not

Copyright protects original expression, not information. A share price, a street address, a court docket number, and a product's weight are facts, and copying facts is not infringement.

The complications start one step up. The selection and arrangement of a compilation can be protectable if the choices involved enough originality, so wholesale duplication of a curated database can infringe even where each individual entry is a bare fact. And anything with authorship in it — reviews, descriptions, photographs, editorial text, code — is protected on its own terms. The U.S. Copyright Office is the reference point for how these categories work.

Fair use, at 17 U.S.C. § 107, is the defense that most technical projects reach for. It is four factors weighed together — purpose and character of the use, nature of the work, amount taken, and effect on the market — and it is decided case by case rather than by category. The Copyright Office's fair use index collects how courts have actually applied those factors, which is more useful than any rule of thumb.

The unresolved question: scraped material as training data

Whether copying copyrighted works to train a machine-learning model is infringement, or is excused as fair use, is not settled law as of mid-2026. Federal district courts issued conflicting decisions on the question during 2025, and the disagreement has not been resolved by a controlling authority.

Anyone telling you the answer is confident beyond what the case law supports. The honest position is that the outcome may vary with how the material was obtained, whether the model reproduces protected expression in its outputs, and what market harm can be shown. This publication does not predict which way it lands.

Two practical consequences follow. Records of provenance — what was collected, from where, under what terms — become the difference between a defensible position and a guess, and those records are exactly what a civil subpoena for documents will ask for. Separately, generated outputs raise their own state-law problems around likeness and voice, an area moving quickly in the state rules on synthetic media.

Common questions

Is scraping a public web page illegal?

There is no single yes or no. Federal access law has been read narrowly for pages served without any login, but that resolves one theory only. The same collection can still breach terms the collector agreed to, burden servers enough to be a tort, copy protected expression, or run into a state statute. Ask all four questions separately rather than treating one favorable ruling as clearance for the whole project.

Does a robots.txt file have legal force on its own?

Robots.txt is a convention, not a statute, and ignoring it is not automatically unlawful. It still matters in practice. It is evidence that the operator communicated a restriction, which can support a claim that access was unauthorized or that conduct was knowing rather than inadvertent. Courts have not treated it as decisive either way, so it is best read as a signal that raises risk rather than as a rule that settles it.

Can I republish scraped data if it is all factual?

Facts alone are not protected by copyright, so reproducing raw data points is generally not infringement. The risks sit elsewhere. Copying the structure of a curated compilation can implicate the original selection and arrangement, republishing accompanying text or images copies expression, and redistributing personal information can trigger state privacy duties independent of copyright. Factual content also does not cure a contract or server-burden problem.

What usually happens after a cease-and-desist letter arrives?

Most disputes stop there. A letter typically asserts terms-of-service breach and unauthorized access, demands deletion, and warns of suit. It has no independent legal effect, but it does two things: it puts the collector on notice, which can make later conduct look knowing, and it starts a paper trail. Preserving logs and stopping automated changes to the collection setup before responding is the ordinary first move.

What to record before a collector runs

The difference between a defensible project and an indefensible one is usually documentation created before collection starts, not arguments assembled afterward.

• The exact URLs and endpoints in scope, and whether each is reachable without any credential.
• A saved copy of the site's terms as they appeared, plus a record of whether an account, API key, or acceptance click was ever involved.
• The robots.txt file as of the collection date, kept with the rest of the record.
• Configured request rates, concurrency limits, and backoff behavior, with the reasoning written down.
• Which fields are bare facts and which contain authored text, images, or code.
• Whether any field identifies a person, and which state privacy regimes could reach that field.
• What the collected material will be used for, since the copyright analysis follows the use as much as the copying.
• Named ownership of the log retention schedule, so records still exist if a dispute surfaces later.

Work through them in that order. Access status first, because it decides whether the hardest federal theory is even in play; contract second, since assent is the fact that most often decides a case; then load, then content type, then personal data, then intended use.