Web Scraping for AI TrainingWeb Scraping for AI Training › Legal basics
Is web scraping for AI training legal?
It depends on what you collect, where you are and what you do with it. Collecting public pages is common, but copyright, text-and-data-mining opt-outs, the EU AI Act's transparency rules, data-protection law such as the GDPR and website terms can all limit how you may use the data to train a model. This is a general overview, not legal advice.
Updated 10 November 2026 by the ScrapeWise team.
Quick picks
To lower risk
Use public, licensed sources
And respect opt-outs.
To stay accountable
Record provenance
Source and date on every record.
For structured public product data
ScrapeWise (our product)
Public pages, with source URLs kept.
Compared
| Area | What it covers | What to do |
|---|---|---|
| Copyright | Reuse of creative works | Prefer licensed or openly licensed sources; get advice for commercial training |
| Text and data mining opt-outs (EU) | Rights holders reserving content from mining | Check robots.txt, AI opt-outs and site terms before collecting |
| EU AI Act | Duties for general-purpose AI providers | Respect opt-outs; document training data sources |
| GDPR | Personal data of people in the EU | Avoid personal data you don't need; have a legal basis |
| Website terms | Contract rules for using a site | Read them; some ban automated collection |
| Access controls | Logins, paywalls, technical blocks | Don't bypass them |
A practical checklist
- Public only
- No logins, paywalls or bypassed blocks.
- Opt-outs respected
- robots.txt, AI crawler blocks and terms checked.
- Personal data minimised
- Removed unless needed and lawful.
- Provenance kept
- URL and date on each record, so data can be traced and removed.
- Advice taken
- A lawyer consulted before commercial model training.
Frequently asked questions
Is web scraping for AI training legal?
It depends on the data, the country and the use. Copyright, text-and-data-mining opt-outs, the EU AI Act, the GDPR and website terms can all apply. Get legal advice for commercial training.
Do I have to respect robots.txt for AI training?
In the EU, machine-readable opt-outs from text and data mining must be respected, and robots.txt is a common way to express them. Respecting it everywhere is good practice.
Does the EU AI Act affect training data?
Yes. Providers of general-purpose AI models must respect copyright opt-outs and publish a summary of the content used for training.
Can I train on public personal data?
Public personal data is still protected by the GDPR in the EU. You need a legal basis, and should remove personal data you don't need.