Agents are good at deciding what to look up and bad at knowing whether what they found is true, current or complete. The Model Context Protocol (MCP) does not fix that by itself. It gives you a standard place to put the fix: tools whose inputs, outputs, freshness and sources are explicit. This guide covers the patterns that work for web data and the design details that separate a reliable tool from a fragile one.
MCP in one paragraph
MCP is an open protocol that lets AI applications discover and call external tools in a standard way. Anthropic introduced it and in December 2025 donated it to the Linux Foundation’s new Agentic AI Foundation, co-founded with OpenAI and Block. In the current specification (revision 2026-07-28), a server lists tools, each with a name, a description, an inputSchema in JSON Schema, and optionally an outputSchema. When a tool declares an output schema, the server must return conforming structuredContent. The spec defines two standard transports: stdio for a local subprocess, and Streamable HTTP for remote servers.
That is enough protocol to build on. The hard decisions are about data.
Why agents get web data wrong
Without a data layer, an agent answering “what does competitor X charge for plan Y?” will search, open a page, read it and answer. Each step can fail quietly:
- The page is cached, outdated or the wrong regional version.
- The page blocks automated traffic. In our AI Crawler Access Index, 99 of the 689 Tranco top 1,000 homepages that responded (14.4%) returned HTTP 403 to a single identified request on 6 Nov 2026.
- The agent extracts the wrong number and nothing checks it.
- The answer carries no source or date, so nobody can verify it later.
Browser agents also remain inconsistent on live sites: on the Online-Mind2Web benchmark, most agents tested completed about 30% of tasks under human evaluation, with the best at 61%. That is fine for exploration and poor for a number someone will act on. For more on that trade-off, see browser agents vs. scrapers.
Two patterns: live-scrape tools and curated-dataset tools
Most web-data MCP servers fall into one of two patterns.
Live-scrape tools fetch and extract a page at call time. Examples include Apify’s hosted server at mcp.apify.com (everydev) and Bright Data’s Web MCP, which launched a free tier of 5,000 requests per month in 2025.
Curated-dataset tools query data that a pipeline has already collected, validated and stored. The agent never touches the source site; it queries a table whose freshness and quality are known. Bright Data also exposes dataset-style tools such as web_data_amazon_product, which sits between the two.
| Live-scrape tools | Curated-dataset tools | |
|---|---|---|
| Best for | Open-ended questions, any URL, one-off lookups | Repeated questions about known entities |
| Latency | A full page fetch and extraction per call | A database or index lookup per call |
| Freshness | Now, if the fetch succeeds | Known: last refresh time per record |
| Consistency | Varies with page layout and blocks | Same schema every call |
| Verification | Usually none at call time | QA before data is served |
| Cost per call | Per request or per page | Mostly fixed by refresh cadence |
| Failure mode | Empty or wrong extraction at answer time | Stale data if the pipeline breaks (detectable) |
The two combine well. A common setup gives the agent curated tools for the core entities it asks about daily (competitor prices, company records, product catalogs) and one general live-fetch tool for everything else, with the description telling the model which to prefer.
When live-scrape tools are enough
If your agent mostly does research, where every question is different and an occasional miss is acceptable, use an existing live-scrape MCP server. Building your own curated layer for that workload is wasted effort. Our comparison of scraping APIs and platforms covers the options.
When you need curated tools
Curated tools pay off when the same entities are asked about repeatedly, when answers feed decisions or customer-facing output, when you need history (“what was the price last month?”), or when the source sites are hard to fetch reliably at call time.
Tool design rules
Whichever pattern you use, these rules make the difference between a tool the model uses correctly and one it misuses.
Keep tools narrow
One tool per question type beats one query_anything tool. get_competitor_price(sku, competitor) is easier for a model to call correctly than a generic search with free-text filters. Narrow tools also make access control, rate limiting and logging simpler. The spec notes that clients aggregating several servers can hit name collisions (two servers each exposing search), so use specific names.
Constrain inputs
Use enums for fixed vocabularies (regions, currencies, competitor IDs), formats for dates, and additionalProperties: false. Every constraint is a mistake the model cannot make. Validate inputs on the server anyway; the spec requires servers to validate all tool inputs.
Declare an output schema
An outputSchema lets clients validate results and tells the model exactly which fields it will get. The spec recommends also returning the serialized JSON as text for backward compatibility.
Return freshness on every value
Include retrieved_at (when the value was observed) and a simple freshness signal, such as stale: true when a record is older than its promised cadence. State the refresh cadence in the tool description. An agent that knows a price is three days old can say so; one that does not will present it as current.
Return the source
Every value should carry the source_url it came from. This lets the agent cite its answer, lets a human verify it in one click, and makes errors traceable to a specific page and extraction. Mark values as observed, derived or estimated; see what a data provenance record should contain.
Make errors actionable
The spec distinguishes protocol errors from tool execution errors returned with isError: true. Use the latter with a message the model can act on: “Unknown competitor ‘acme’. Valid values: acme_corp, acme_eu.”
An illustrative tool definition
The definition below is illustrative only, not a published API. It shows the fields discussed above in one place.
{
"name": "get_competitor_price",
"title": "Competitor price lookup",
"description": "Latest validated price for one of our SKUs at one competitor. Data refreshes daily; records older than 36 hours are flagged stale. Read-only.",
"inputSchema": {
"type": "object",
"properties": {
"sku": { "type": "string", "pattern": "^SKU-[0-9]{4,6}$" },
"competitor": { "type": "string", "enum": ["competitor_a", "competitor_b"] },
"region": { "type": "string", "enum": ["US", "CA", "GB"] }
},
"required": ["sku", "competitor", "region"],
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"properties": {
"price": { "type": "number" },
"currency": { "type": "string" },
"price_type": { "type": "string", "enum": ["regular", "sale"] },
"availability": { "type": "string" },
"match_confidence": { "type": "number" },
"source_url": { "type": "string", "format": "uri" },
"retrieved_at": { "type": "string", "format": "date-time" },
"stale": { "type": "boolean" }
},
"required": ["price", "currency", "source_url", "retrieved_at", "stale"]
}
}Notice what the description does: it states the cadence, the staleness rule and that the tool is read-only. The model reads descriptions when deciding what to call, so put operating facts there.
Hosting and auth
Local (stdio). The client launches your server as a subprocess. It is simple and good for a single developer or a desktop app. Per the authorization spec, stdio servers should read credentials from the environment rather than run an OAuth flow.
Remote (Streamable HTTP). Needed when several users, agents or products share the server. Authorization is optional in MCP, but when an HTTP server uses it, the spec bases it on OAuth 2.1: the server acts as a resource server, must publish OAuth Protected Resource Metadata (RFC 9728), and must validate that each access token was issued for it. Tokens go in the Authorization header, never in the query string. Request scopes by least privilege, for example read access per dataset.
Operational points that matter more than the protocol:
- Rate limit and log every call. The spec lists rate limiting among server requirements; logs also show which tools the model actually uses.
- Keep data tools read-only. A tool that only reads is far easier to reason about when a model calls it unexpectedly.
- Treat scraped text as untrusted. Web pages can carry prompt-injection payloads. Anthropic notes that even a 1% attack success rate “still represents meaningful risk”. Return extracted fields instead of raw page text where you can, and sanitize what you return.
- Cache with intent. For curated tools, the cache is the dataset. For live tools, cache briefly and expose the cache age.
- Monitor the pipeline, not just the server. An MCP server can be up while the data behind it is a week old. Alert on freshness, not only on uptime.
Agent-ready data, delivered
If your agents need the same web data every day and you want it validated, fresh to a known date and sourced, our agent-ready data service runs the pipelines and serves the results through a hosted MCP server or API (+$300 per month, included in the Team bundle and above; see pricing). If a general live-scrape server already covers your use case, use that instead. To see what curated tools would return for your sources, request a free 100-row sample and get it within 2 business days.