What teams build with a web scraper that remembers

The index is a snapshot and the docs keep moving. A deprecated flag keeps being recommended, a deleted page keeps being retrieved, and nobody sees it because a stale embedding scores as well as a fresh one. Full re-indexing is the fix teams reach for, and it is the one they cannot afford to run every night.

{
  "seed": "https://docs.example.com/",
  "schedule": "daily",
  "config": {
    "include_paths": ["/docs/*"],
    "only_main_content": true,
    "sitemap_incremental": true
  }
}
// and a destination on the project:
{
  "provider": "qdrant", "target": "docs",
  "mode": "upsert", "key": "url",
  "columns": ["url", "title", "markdown", "content_hash", "change_class"]
}

Incremental runs read only what the sitemap says moved. The destination is incremental too: only added, modified and removed pages reach the index, chunked and embedded on your own key.

A guidance page changes on a Tuesday and you read about it in a newsletter on Friday. A consultation closes and the page that said so is gone. Page-monitoring tools watch one URL at a time, alert on a rotating banner, and cannot tell a page that was deleted from a page that was blocked for a crawler.

{
  "seed": "https://www.gov.example/",
  "schedule": "daily",
  "config": {
    "crawl_mode": "sitemap_first",
    "include_paths": ["/guidance/*", "/news/*"],
    "sitemap_incremental": true,
    "notify_min_words": 20,
    "chat_url": "https://hooks.slack.com/services/…",
    "change_digest": { "connection_id": "…", "focus": "deadlines, forms, thresholds" }
  }
}

Two sections of a large site, from its sitemap; an incremental run touches only what lastmod says changed. The record goes to the channel the team already watches, with a paragraph on what it means.

Pricing, feature and changelog pages change without an announcement. The tools that watch them alert on the navigation bar and go quiet on the number that mattered, and reading forty diffs to find the one price change is nobody’s Monday.

{
  "seed": "https://competitor.example/",
  "schedule": "daily",
  "config": {
    "include_paths": ["/pricing", "/changelog/*", "/docs/*"],
    "only_main_content": true,
    "json_schema": [
      { "name": "plan_names", "type": "array" },
      { "name": "monthly_prices", "type": "array" }
    ],
    "llm_extract": { "connection_id": "…" },
    "change_digest": { "connection_id": "…", "focus": "prices, plan limits, new features" },
    "chat_url": "https://discord.com/api/webhooks/…"
  }
}

Three sections of one site. The fields are filled by your model only for pages whose content changed; the digest is one paragraph, ranked by what you said you care about.

Client sites change under you; a rival’s blog publishes twice a week; a migration drops forty pages and the 404s are found when the traffic is. Crawlers that audit a site once do not remember the site the week before.

{
  "seed": "https://client.example/",
  "schedule": "weekly",
  "config": {
    "include_paths": ["/blog/*"],
    "formats": ["markdown", "links"],
    "chat_url": "https://hooks.slack.com/services/…"
  }
}
// before a migration: run; after cut-over: run again — the record is the audit.

One section, weekly. The head fields — title, description, canonical, robots, h1 — are read on every page and diffed on every run without being asked for.

An assistant cannot read a docs page, a pricing page or a supplier’s spec sheet — and when it can, the page eats the context window and the assistant reasons over navigation. Ask it to set up monitoring and it guesses at configuration keys.

pip install "mesharc[mcp]"
claude mcp add mesharc -e MESHARC_API_KEY=mesharc_… -- mesharc-mcp

# then, in the assistant:
you     Map fastapi.tiangolo.com and tell me which pages are about security.
you     Set up a project for hotspotseo.com that only scrapes the blog, weekly.
you     What changed on it since last week?

Seventeen tools. Mapping, searching, listing and changing settings cost nothing; a page costs what it took to read. Hosted, you approve the app and revoke it like any key; run locally, the key never leaves your machine.

A directory, a catalogue, a listings site has to become a table and stay current. The scraper that fills it is a pet: it breaks on a layout change, it re-fetches what did not change, and the rows it writes are whatever it found, not the shape the query wants.

{
  "seed": "https://catalogue.example/",
  "schedule": "weekly",
  "config": { "include_paths": ["/products/*"], "json_schema": [{ "name": "price", "type": "number" }, { "name": "sku", "type": "string" }] }
}
// the destination, with its shape:
{
  "provider": "postgres", "target": "products", "mode": "upsert", "key": "url",
  "columns": ["url", "title", "fields", "content_hash", "crawled_at", "change_class"],
  "shape": {
    "filter": [{ "field": "fields.price", "op": "exists" }],
    "map": [{ "name": "price_cents", "from": "fields.price", "ops": ["number", "mul:100"], "cast": "int" }],
    "drop": ["fields"]
  }
}

Fields come from what the page declares — JSON-LD, Open Graph, microdata — with a model on your key for the rest. The shape is declarative and previewed against the last run before it is saved.