Is scraped web data still useful for training AI models?

For general pretraining, free corpora like Common Crawl and large vendor archives already exist, and frontier labs increasingly license content directly. Web data is most useful for narrow domains, specific languages, evaluation sets and retrieval corpora.

What is eval contamination?

Contamination happens when test items, or text very close to them, appeared in a model's training data, so the model can score well by recall instead of skill. Web-sourced eval items are especially exposed because crawlers may have collected the same pages.

How do I keep a RAG knowledge base fresh?

Re-crawl sources on a cadence matched to how often they change, detect changed and deleted documents, re-embed only what changed, and store retrieved_at on every chunk so the agent can tell how old its context is.

How should licensing and opt-out signals be handled in a dataset?

Record them per document as technical metadata: robots.txt status for relevant crawlers, Content Signals, any RSL terms, and the retrieval date. That lets the team using the data apply its own policy. It is not a substitute for legal advice.

When should I go to a human data company instead?

When you need expert-written answers, preference rankings or reasoning traces from domain specialists. That is a human labor problem, and firms such as Mercor, Surge and Scale are built for it.

Building eval and RAG datasets from web data

“Training data” covers several very different businesses. Before deciding whether web data belongs in your AI project, it helps to see where the money in the category actually goes, because it shows which problems are already solved, which are labor problems, and which are data engineering problems that web collection fits.

Where the money in training data actually is

The largest flows are not for scraped pages.

Expert human data. Mercor reportedly booked about $614M in gross revenue in the first half of 2026 and reached about a $2B annualized gross run rate by June 2026, according to secondary reports. Surge AI passed $1B in revenue in 2024, according to Sacra. Meta paid $14.3B for 49% of Scale AI in 2025, according to HeroHunt. These companies sell specialist labor: experts writing answers, ranking outputs and grading reasoning.
Licensed content. Labs are paying publishers and platforms directly. Wikimedia Enterprise signed Microsoft, Meta, Amazon, Perplexity and Mistral in January 2026, as reported by Ars Technica.
Raw web at scale. Common Crawl remains the free baseline, and Bright Data claims an archive of more than 200B HTML pages growing by 15B a month, with major AI labs as customers.

The conclusion is plain. If you need generic web text for pretraining, it already exists. If you need expert judgment, you need experts, not a crawler. A web-data partner, us included, is the wrong vendor for both.

Where web data does fit

Web collection earns its place where the data is specific, structured, current, or in a language the big corpora cover thinly.

Use	What it looks like	Why web data fits
Niche domain corpora	All public product manuals in a category, regulatory filings in one sector, technical documentation for a toolchain	Thin coverage in general crawls; needs targeted collection and cleaning
Multilingual corpora	Domain text in languages under-represented in general crawls	English is about 42.6% of pages in Common Crawl’s latest crawl, CC-MAIN-2026-43, and no other language tops 6.5% (Common Crawl statistics)
Eval sets	Questions with answer keys tied to a source URL and date	Real-world facts make realistic tests, and sources make answers checkable
RAG knowledge feeds	Continuously refreshed documents, chunked and tagged, feeding an internal agent	Freshness and change detection are pipeline problems, not model problems
Structured reference data	Entities with attributes (companies, products, locations) for grounding and tool calls	Needs schema stability and refresh, which is what managed feeds do

Building an eval set from web data

A good eval set is small, sourced, versioned and private. A workable process:

Define the task precisely. “Extract the warranty period from a product page” is testable. “Understand products” is not.
Sample sources deliberately. Cover the layouts, languages and edge cases your system will meet in production, not just the easy pages.
Draft items, then review them. Models can draft question-and-answer pairs from source documents quickly. A person should check every answer key against the source. An unchecked key turns your eval into a measure of agreement with another model.
Store the evidence. Every item carries its source_url, retrieved_at, the supporting passage, and a snapshot hash so the item can be re-verified later.
Flag time-sensitive answers. Prices, officeholders and counts change. Mark those items with a valid_as_of date and refresh or retire them on a schedule.
Version and hold out. Freeze versions, keep a private held-out split that never leaves your environment, and record which model versions were tested on which version.

Contamination

Web-sourced eval items are exposed to contamination: the same page may already sit in a model’s training data, so a high score can reflect recall rather than capability. Mitigations that work in practice:

Prefer recent sources. Record the publication date of each source and favor content published after the training cutoff of the models you test.
Check overlap. Run n-gram or near-duplicate checks between eval items and the corpora you can access, including your own training and RAG data.
Transform, do not just copy. Questions that require combining facts from several sources, or applying a fact to a new case, are less likely to be answerable from memory than verbatim lookups.
Keep it private and refresh it. A public eval set will eventually be crawled. Rotate in fresh items each version.

Building RAG knowledge feeds

RAG quality depends less on the embedding model than on whether the documents are current, clean and attributable.

Freshness per document. Store retrieved_at on every chunk and return it with retrieved context, so the agent can say how old its information is.
Change detection. Re-crawl on a cadence matched to each source, compare normalized content, and re-embed only what changed. This keeps cost proportional to change, not to corpus size.
Deletes matter. When a source page disappears or is replaced, tombstone its chunks. Stale chunks from removed pages are a common source of confident wrong answers.
Canonical URLs. Resolve redirects and duplicates to one canonical URL per document, so citations are stable and the same text is not indexed three times.
Boilerplate removal. Strip navigation, cookie banners and footers before chunking. Otherwise retrieval matches on the template rather than the content.

Quality and dedup

For both corpora and feeds, the same cleaning stack applies:

Step	Method	Catches
Exact dedup	Hash of normalized text	Mirrors, syndicated copies, URL variants
Near-dup	MinHash or similar shingling	Templated pages differing in a few words
Language ID	Per-document and per-paragraph detection	Mixed-language pages, mislabeled content
Quality filters	Length, symbol ratios, extraction completeness	Error pages, empty renders, spam
Extraction QA	Schema validation, sampled human review	Silent parser failures
Model-assisted review	A second model checks a sample of outputs	Systematic errors in LLM-produced fields

Sampled review is worth measuring, not just doing. On Nomad Watch, the public dataset we maintain with an agent pipeline, a sampled cross-model judge check showed batch-3 pass rates of 100% for city scores, 87% for guides and 82% for costs. Earlier cost batches passed only 55–64% and were sent back for fixes. Without the sample, those weak batches would have shipped looking identical to strong ones. Nomad Watch also labels its city scores and costs as LLM-generated estimates, separate from its source-backed embassy and treaty records, which is the same observed-versus-estimated distinction a training or eval set needs. See what a data provenance record should contain.

Licensing signals as technical metadata

Sites now publish machine-readable preferences about AI use, and a dataset should carry them per document so the team using it can apply its own policy. Signals worth recording at retrieval time:

robots.txt rules for the relevant crawlers. In our AI Crawler Access Index of the Tranco top 1,000 domains (6 Nov 2026), 510 served a valid robots.txt; of those, 151 (29.6%) fully blocked at least one AI training crawler, and 89 (17.5%) fully blocked at least one AI search crawler.
Content Signals, Cloudflare’s robots.txt extension with search, ai-input and ai-train values. Adoption is still low in our index: 17 of the 510 valid robots.txt files (3.3%) contained them.
RSL terms. RSL 1.0, released in December 2025, puts licensing and payment terms in a machine-readable form.
IETF AIPREF, a vocabulary for AI usage preferences that was still in draft as of November 2026.

Record these with the retrieval date, because they change. Treat them as facts about the source at a point in time, and leave the policy decision to the people accountable for the model. This is operational metadata, not legal advice.

When not to use a web-data partner

You need expert-written answers, preference data or reasoning traces: go to a human data firm.
You need broad web text for pretraining: start with Common Crawl or an existing archive.
You need a one-time corpus from a few cooperative sites and have an engineer free for a week: build it yourself.

A partner makes sense when the corpus is niche or multilingual, the eval set needs sourced and re-verifiable answer keys, or the RAG feed has to stay fresh month after month without your team maintaining crawlers.

Custom datasets for eval and RAG

Our custom datasets service builds domain corpora, eval sets and refreshed RAG feeds with a source URL and timestamp on every value and a QA gate before delivery. A Data Sprint starts at $1,500, and recurring feeds are priced per source (see pricing). If you have a target domain or language in mind, request a free 100-row sample and get it within 2 business days.