Web Scraping for AI TrainingWeb Scraping for AI Training › Data formats
What format should AI training data be in?
Use JSON Lines (one JSON record per line) for text datasets up to a few million records, and Parquet for very large or tabular datasets. Keep raw pages in WARC if you need to reprocess them later. Whatever the format, store the source URL, collection date and language with every record.
Updated 10 November 2026 by the ScrapeWise team.
Quick picks
For text datasets
JSON Lines
The common default.
For large or tabular sets
Parquet
Smaller and faster.
For structured product data
ScrapeWise (our product)
Exports to CSV, Excel or API.
Compared
| Format | Best for | Size | Note |
|---|---|---|---|
| JSON Lines (.jsonl) | Text and instruction datasets | Medium | One record per line; easy to stream |
| Parquet | Large or tabular datasets | Small (compressed) | Fast column reads |
| CSV | Simple tables, spreadsheets | Medium | Fragile with long text |
| WARC | Raw web pages for reprocessing | Large | Used by Common Crawl |
| ScrapeWise exportsOur product | Structured product and price data | — | CSV, Excel, REST API or MCP |
ScrapeWise exports
Rows include fields such as name, price, currency, brand, EAN and source URL.
Metadata to keep on every record
- Source URL
- Where the content came from.
- Collected at
- Date and time of collection.
- Language
- Detected language of the text.
- Licence or permission note
- What you know about reuse rights.
- Content hash
- To find duplicates later.
Frequently asked questions
What format should AI training data be in?
JSON Lines for most text datasets, Parquet for very large or tabular data, and WARC for raw pages you may reprocess.
Is JSONL or Parquet better?
JSONL is simpler and easy to stream; Parquet is smaller and faster for large datasets. Many teams write JSONL and convert to Parquet.
Should I keep the raw HTML?
If you may need to re-extract later, keep raw pages, for example in WARC; otherwise keep the cleaned text plus the source URL.
What metadata should training data include?
At least the source URL, collection date and language; ideally also a licence note and a content hash.