The problem
A useful city guide needs dozens of facts per city: costs, safety, internet, climate, visas, taxes, embassies. Multiply by
3,000 cities and 250 countries and you have hundreds of thousands of fields that go stale. Nobody maintains that by hand.
A single agent asked to "research Lisbon" will produce something plausible, but plausible is not the same as checked.
The system
We split the work into small tasks and put a gate between the agents and the published data. Every stage leaves a trace,
so any value on the site can be traced back to a run, a source and a reviewed change.
What it produced
Between 9 October 2026 and 6 November 2026, the queue recorded 7,068 agent runs across 5 model backends
from different vendors, and completed 3,463 research tasks:
The published dataset holds 3,169 city records, 6,794 embassy records and 3,625 tax treaties,
with roughly 39,600 dated source citations in total. It's stored as plain YAML files in version control, one file
per record, so every change has a diff and an author.
Where the gate said no
The most useful numbers in this case study are failures. The judge samples 50 records per dataset per batch and checks
them against the sources. In the first batch it confirmed only 64% of cost records and 76% of city scores, both below the bar.
In the second batch costs confirmed at just 55%. Those batches did not ship as they were: each went through fix passes and was
re-checked first. The third batch passed on the first check, at 100% for city scores,
87% for guides and 82% for costs. As we write this, the fourth batch's cost records
sit at 67% and are back in a fix pass.
In every sampled batch the judge found zero invented values. The misses were facts that didn't hold up closely enough against
the source, not fabrications. That distinction matters, and you only know it because something is checking.
Without the gate, the first cost batches would have gone live looking perfectly reasonable. That is the core risk of agent-generated
data, and the reason we never let a model's output reach a customer unchecked.
What we're careful to say
- Scores and cost levels are estimates. City scores, cost levels and guide texts are researched and synthesized by agents and labelled as generated estimates on the site. They are checked, not measured.
- Embassies and tax treaties are source-backed. Each of those records cites the official source it came from.
- Client feeds are stricter. For customer feeds, values are extracted from pages, not synthesized. Derived fields are labelled as derived.
What this means for your data
The same building blocks run client feeds: small scoped tasks, agents that write and repair extractors, a deterministic gate,
an independent model check, reviewed changes and a full trail from value to source. The difference is that a customer feed reads
values off pages instead of researching them, which makes the checks stricter, not looser.
Nomad Watch is also a public site in its own right, which gives the data a real audience: 15.6k visitors in the
90 days to July 2026 and 8,997 clicks from Google search.