What a data provenance record should contain

A dataset without provenance asks you to trust it. A dataset with provenance lets you check it. That difference matters more now that much of the data flowing into dashboards, CRMs and AI agents passes through a language model somewhere along the way. When a number looks wrong, the first question is always the same: where did this come from? A provenance record is the answer, written down before anyone asks.

This guide lists what that record should contain, shows an example, and explains how we apply it on a real dataset.

Why provenance is a data quality feature

Provenance is often framed as paperwork. In practice it is what makes data debuggable and usable.

• Analytics. When a metric shifts, provenance tells you whether the world changed or your extractor did. Without it, every anomaly turns into an investigation.
• AI agents and RAG. An agent that can return the source_url and date behind an answer can cite it. One that cannot is asking the user to take its word for it. See how to give your AI agent reliable web data with MCP.
• Repair. When you fix a parsing bug, provenance tells you exactly which records the old extractor version touched, so you can reprocess those and nothing else.
• Trust between teams. A buyer, analyst or downstream engineer can decide how much weight to give a value when they can see whether it was read from a page or estimated by a model.

The fields

Here is the checklist, roughly in order of importance.

You do not need every field on day one. If you implement only four, make them source_url, retrieved_at, method and kind.

Observed, derived and estimated

This distinction deserves its own section because it is the one most often blurred.

• Observed: read directly from a source. A price on a product page, an address in a registry, a date in a press release.
• Derived: computed deterministically from other values. A unit price from price and pack size, a currency conversion, a sum.
• Estimated: produced by a model or heuristic without a direct source. An LLM’s guess at a company’s headcount, a predicted cost of living, a classification.

All three can be useful. Mixing them without labels is how an estimate gets quoted as a fact two hops downstream. Label the kind on every value, and when a model fills a gap, say so in the data, not in a footnote.

Record-level vs. field-level

If one page produces one record in one pass, record-level provenance is enough. Once you enrich, merge sources or fill gaps with models, fields within the same record have different origins, and you need field-level provenance. A common compromise: record-level provenance for the primary source, plus a field-level override map for any field that came from somewhere else.

An example record

Here is a field-level example for a company record that combines a website extraction, a registry lookup and a model estimate:

{
  "entity_id": "co_01J9Z8K2",
  "fields": {
    "legal_name": {
      "value": "Example Logistics Inc.",
      "kind": "observed",
      "source_url": "https://registry.example.gov/companies/123456",
      "retrieved_at": "2026-11-04T08:12:55Z",
      "method": "api",
      "extractor": "registry_v3",
      "extractor_version": "a41c9e2",
      "confidence": 1.0
    },
    "warehouse_count": {
      "value": 7,
      "kind": "observed",
      "source_url": "https://example-logistics.com/locations",
      "retrieved_at": "2026-11-05T14:03:10Z",
      "valid_as_of": null,
      "method": "rendered_browser",
      "extractor": "locations_page",
      "extractor_version": "f02b7d1",
      "snapshot_hash": "sha256:9b1f…",
      "confidence": 0.92,
      "review": { "lint": "pass", "judge": "pass", "human": null },
      "history": [
        { "value": 6, "retrieved_at": "2026-08-02T10:41:00Z" }
      ]
    },
    "employee_band": {
      "value": "201-500",
      "kind": "estimated",
      "method": "llm_generated",
      "derived_from": ["warehouse_count", "legal_name"],
      "extractor": "headcount_estimator",
      "extractor_version": "prompt-2026-10-21",
      "confidence": 0.55,
      "review": { "lint": "pass", "judge": "pass", "human": null }
    }
  }
}

Three things to notice. The estimated field has no source_url, and it does not pretend to. The observed fields carry the exact page and extractor version that produced them. And history shows the warehouse count moved from 6 to 7 between August and November, which is often the most valuable signal in the record.

Change history

Store history as append-only observations rather than overwriting values. Each observation is a row: entity, field, value, retrieved_at, source_url, extractor_version. The “current” record becomes a view over the latest observation per field.

Two practical rules:

• Record unchanged checks too, at least as a last_confirmed_at timestamp. “Still 7 as of yesterday” is different from “7 three months ago.”
• Separate a change in the world from a change in extraction. If a value changes at the same moment the extractor version changes, flag it for review before treating it as real.

How we apply this: Nomad Watch

Nomad Watch is a public dataset we maintain with an agent pipeline (case study). It holds about 3,169 city records, each with 70 score factors, 45 cost factors and 86 guide factors, plus 6,794 embassies and 3,625 tax treaties. Between 9 Oct and 6 Nov 2026 the pipeline ran 7,068 agent runs and completed 3,463 research tasks across five model backends.

Provenance is built into the workflow rather than added afterward:

• Kind is labeled. City scores and costs are shown as LLM-generated estimates. Embassies and treaties are source-backed. The site does not blur the two.
• Every run is reviewable. Each run lands as one reviewable pull request, so every change has a diff and a traceable origin.
• Review status is recorded. Every run passes a deterministic lint gate, and a sample goes through a cross-model “judge” check. In batch 3, judge pass rates were 100% for city scores, 87% for guides and 82% for costs. Earlier cost batches passed only 55–64% and were sent back for fixes.

Those pass rates are only knowable because the review results are recorded alongside the data. Without that, a weak batch would have looked the same as a strong one.

Storage and cost

Provenance adds columns, not complexity, if you plan for it:

• Field-level provenance roughly multiplies row width. Columnar formats such as Parquet compress the repeated values (method, extractor version) well.
• Raw snapshots are the expensive part. Keep hashes forever, full snapshots for a fixed retention period, and longer only for high-value sources.
• If you want a formal model, W3C PROV (2013) describes provenance in terms of the entities, activities and people involved in producing data. Most teams borrow its concepts rather than implement it in full.

A short checklist

Before you ship a dataset or connect it to an agent, check that you can answer yes to each of these:

• Can someone open the exact page behind any observed value in one click?
• Does every value say when it was retrieved, and, where the source states one, the date it applies to?
• Can you tell which extractor version produced a value, and reprocess only those records after a fix?
• Is every value labeled observed, derived or estimated, with no estimate presented as a sourced fact?
• Do derived values point to their inputs?
• Is it recorded which checks each value passed, including sampled human or model review?
• Can you show a value’s previous states and when each was seen?
• Can you re-extract from a stored snapshot, or at least prove what the page contained via its hash?

If the answer to the first two is no, fix those before anything else. They cost little to capture at collection time and are nearly impossible to reconstruct later.

When you can skip most of this

For a one-off list you will use once and throw away, a source_url and a date column are enough. Full provenance earns its cost when data is refreshed, merged, fed to agents, or relied on by people who did not collect it.

Provenance by default

Every dataset we deliver through custom datasets carries a source URL and timestamp on every value, and the rest of this checklist (method, extractor version, kind, review status, history) can be part of the schema we agree with you. Data Sprints start at $1,500 (see pricing). If you want to see what that looks like for your sources, request a free 100-row sample and get it within 2 business days.