How to monitor competitor prices at scale

Fetching a price from a product page is the easy 10% of competitor price monitoring. The other 90% is knowing that the number you fetched belongs to the same product as yours, in the same pack size, region and currency, that it is the price a customer actually pays, and that it changed when your system says it changed. This guide walks through each of those problems in the order you hit them.

What one price observation should contain

Before writing any scraper, define the record. A bare price column is the root of most bad pricing decisions. A usable observation looks like this:

If your monitoring tool cannot give you most of these fields, you are getting a number, not a price.

Step 1: Match the catalog before you scrape it

SKU matching decides whether the rest of the pipeline is worth anything. Work down a ladder of match methods and record which one produced each pair.

1. Global identifiers. GTIN (EAN/UPC) is the strongest key. Many retailers embed it in structured data on the product page; schema.org’s Offer type defines gtin, gtin13, mpn and sku alongside price and priceCurrency. When present, this is both the easiest and the most reliable field to read.
2. Brand plus manufacturer part number. Almost as good, once you normalize hyphens, spaces and case.
3. Title and attributes. Fuzzy matching on normalized titles plus key attributes (size, capacity, color, pack count). This is where LLMs help: they are good at judging “is this the same product?” from two listings. Treat their output as a candidate with a confidence score, not a fact.
4. Human review. Route low-confidence matches to a person. A reviewed match is an asset; store it so you never pay to decide it again.

Re-verify matches periodically. Retailers reuse URLs for new models, and a page that matched last quarter may now sell the successor product.

For private-label goods and bundles there is no true match. Decide upfront whether you monitor “nearest equivalent” products, and label them as such so nobody reads them as like-for-like.

Step 2: Set cadence by tier, not globally

Refresh frequency is the biggest cost driver, because cost scales with pages multiplied by checks. Tier the catalog:

These are starting points, not rules. Measure how often prices actually change per competitor and category for a few weeks, then move SKUs between tiers. A competitor that reprices weekly does not need hourly checks.

Step 3: Handle variants, packs and units

Variant handling is where many tools quietly go wrong. Common traps:

• Default variant. The page loads with one size or color selected. Your scraper reads that price, but you sell a different variant.
• Price per selection. The price only appears after choosing options, often through a JavaScript call. You may need to call the variant endpoint the page uses, or render the page.
• Pack sizes. A 6-pack at $29.94 and a single at $5.49 need a unit_price to be compared at all.
• Bundles and add-ons. “Includes free case” changes the value without changing the price field.

Model variants explicitly in the record and match at variant level wherever you price at variant level.

Step 4: Currency, regions and taxes

The same URL can show different prices depending on where the request comes from, which store is selected, which cookies are set and whether the visitor is logged in. Record the request context with every observation: country or region, store or postal code if relevant, and whether displayed prices include tax.

Store the observed price in the observed currency. If you need a common currency, convert in a separate column with the exchange rate and its date. Never overwrite the observed value with a converted one; you lose the ability to tell a competitor’s price change from an exchange-rate move.

Monitor only what an ordinary logged-out visitor sees unless you have an agreement with the site. Logged-in and member pricing is a different access question, and usually not worth the trouble.

Step 5: Detect changes you can trust

Raw diffs between runs produce noise. A good change-detection layer:

• Compares normalized values, not page text, so a redesign or reformatted number does not register as a price change.
• Re-checks before alerting. When a price moves more than a threshold, fetch the page again a few minutes later from the same context. Many false alerts disappear here: A/B tests, cache artifacts, and partial page loads.
• Separates price types. A sale price appearing over an unchanged list price is a promotion. A list price change is a repricing. Most teams want different alerts for each.
• Tracks stock transitions. “Out of stock” to “in stock” at a lower price is often the most actionable event in the feed.
• Keeps history append-only. Store every observation, not just the latest value, so you can answer “when did they start undercutting us?” months later.

Step 6: Plan for access controls

Retail sites increasingly gate automated traffic. In our AI Crawler Access Index, we sent one plainly identified homepage request to each of the Tranco top 1,000 domains on 6 Nov 2026. Of the 689 homepages that responded, 99 (14.4%) returned HTTP 403, 169 (24.5%) were served through Cloudflare, and 37 answered with a challenge. Those are homepage results; product and search pages on the same sites may be gated differently.

Cloudflare also changed its defaults in 2026: from 15 Sep 2026, bots it classifies as Training or Agent traffic are blocked by default on ad-displaying pages of new domains. That matters if you planned to monitor with a browser agent.

Practical rules that keep monitoring stable:

• Prefer the cheapest reliable source first: structured data in the page, the site’s own JSON endpoints, public product feeds or affiliate feeds, then rendered pages.
• Keep request rates modest and spread checks over the day. Price monitoring rarely needs bursts.
• Read robots.txt and the site’s terms as operational inputs. They tell you what the site expects, and ignoring them tends to end in blocks.
• Expect breakage. Budget for repairs when a retailer redesigns, not as an exception.

Step 7: QA is the product

Scrapers rarely fail loudly. They fail by returning plausible wrong numbers. The QA checks worth automating:

The manual audit matters more than it looks. A weekly sample of a few dozen rows per site, compared by a person against the live page, is the only check that catches errors your rules did not anticipate.

Step 8: Deliver where decisions get made

Pick delivery by who acts on the data:

• Pricing analysts: a warehouse table or daily file (CSV or Parquet) with full history.
• Repricing engines: an API or webhook with only validated changes.
• Category managers: a short digest of meaningful moves, not a dashboard of every SKU.
• AI agents: an MCP tool or API that returns the latest validated price with its source_url and retrieved_at, so the agent can cite it. See how to give your AI agent reliable web data with MCP.

Build vs. buy

Be honest about scale. If you track under a couple of hundred SKUs on a few cooperative sites and a missed day costs nothing, a marketplace actor or a small script is the right answer, and you should not pay a service. The calculus changes when matching is hard, sites block automated traffic, prices feed directly into repricing, or the person who wrote the scraper has moved on. Our comparison of managed feeds and DIY agents goes deeper on that line.

Getting a monitored price feed

If you would rather hand off the matching, cadence, change detection and QA, our web monitoring service runs competitor price feeds with a source URL and timestamp on every value, change alerts, and delivery as files, API or an MCP server, with pricing from $400 per month for monitoring and $600 per source per month for a full managed feed (see pricing). The quickest way to judge whether it fits is to send us a few competitor URLs and get a free 100-row sample within 2 business days.