Website change monitoring API: from diff to Slack to Postgres

Most website change monitoring starts as one URL and a screenshot comparison, and ends as a Slack channel that everyone has muted. The banner rotated, the "last updated" stamp ticked over, a page timed out and was reported as deleted. The alert fired every time, and nobody read it.

What a developer usually wants is narrower and more useful: watch a whole section of a site, hear about real edits only, and have every version of every page land in a table you can query next quarter. This post builds that pipeline with MeshArc in three stages — the diff, the alert, the table — and is clear about where other tools do it better.

Search volume for "website change monitoring" sat at 260–1,000 a month in the US over the last year. The people searching are mostly compliance teams, competitive-intelligence analysts and anyone who has been caught out by a silent edit to a supplier's terms.

What a change monitor has to get right

Four things separate a monitor you trust from one you mute:

1. Coverage. A regulator's guidance library or a competitor's docs is hundreds of pages, and new pages matter as much as edited ones. Watching ten hand-picked URLs misses the page published on Tuesday.

2. Noise. Consent banners, rotating widgets and timestamps change on every load. If they reach the diff, every page is "modified" every day.

3. Refusal is not removal. A page that returned a block, a timeout or a captcha this run has not been deleted. A monitor that cannot tell the difference will one day report half a site as gone.

4. History. An alert tells you something changed today. A table tells you what the page said in March.

Stage 1: the diff

In MeshArc a watched site is a project: a seed URL, a scope and a schedule. The first run sets the baseline; every later run produces a change record comparing each page with its previous version.

pip install mesharc
# or: npm install mesharc

from mesharc import MeshArc

arc = MeshArc()  # reads MESHARC_API_KEY

project = arc.projects.create(
    "https://www.example.gov/guidance",
    name="Regulator guidance",
    schedule="daily",                       # manual | hourly | daily | weekly
    config={
        "max_pages": 300,
        "include_paths": ["/guidance/*"],
    },
)

arc.runs.start(project["id"], wait=True)    # run 1: the baseline

# ...the next scheduled run (or another runs.start) produces the change record
record = arc.changes(project["id"])
print(record["change"]["counts"])
# {'added': …, 'removed': …, 'modified': …, 'withheld': …}

diff = arc.page_diff(project["id"], "https://www.example.gov/guidance/fees")

Each page in the record is classed as added, removed, modified, same, withheld, unverified or skipped. Three behaviours do most of the work:

• Noise is stripped before comparison. Consent banners, rotating widgets and "last updated" stamps are removed first, so they never reach the diff.

• Withheld is not removed. A page the site refused this run is withheld, never removed. And if a run reaches under 90% of the site, removals are held back altogether and the reason is stated, so an interrupted crawl is not read as a mass deletion.

• Fields, not just text. Alongside the word-level diff you get field-level changes to titles, dates, canonicals and prices. A field that suddenly goes empty on every page is raised as a selector break rather than reported as a thousand edits.

Coverage comes from the sitemap. MeshArc walks robots.txt, well-known paths and sitemap indexes, so a page that appears in the sitemap is flagged as added on the run after it is published. On a gov.uk discovery run it mapped 508,290 URLs from 29 sitemap files, which is a useful upper bound on "too many pages to watch by hand".

What it costs

Change records, field diffs, webhooks, chat alerts and destinations cost 0 credits. You pay for pages read, and the billing rules remove most of a monitor's re-reads:

• When the sitemap's lastmod shows a page has not changed, its previous copy is carried forward at no charge.

• Refused pages, 404s and 410s are free, whatever the ladder spent finding out.

• A plain page is 1 credit; a page that needs a real browser is 4.

The worst case for the project above — all 300 pages re-read by plain fetch every day — is about 9,000 credits a month. The Starter plan includes 100,000 for $29, and the free plan's 1,000 credits are enough to set a baseline and run a few diffs on a section that size. Full figures are on pricing.

Stage 2: the alert

There are two ways to hear about a change, and most teams want both.

Slack, Teams or Discord: one URL

Set chat_url in the project's config to an incoming-webhook URL. Slack, Discord, Microsoft Teams (Workflows), Mattermost and Google Chat are supported, and each gets a message in its own format; Slack gets Block Kit.

arc.projects.update(project["id"], config={
    "chat_url": "https://hooks.slack.com/services/T000/B000/XXXX",
    "notify_min_words": 20,   # ignore edits smaller than this
})

notify_min_words is the threshold that keeps the channel readable: a modified page is reported only when its edit reaches that many words. A typo fix stays in the change record but not in Slack.

Chat messages are unsigned, which is how incoming webhooks work, and the API returns the URL masked (https://hooks.slack.com/services/••••) so it does not leak through a config dump.

If you would rather read a paragraph than a list, configure change_digest with a model connection and a focus ("fee changes and new consultation deadlines"). The run.finished event then carries a summary, the key points and a significance flag, written on your own model key.

A signed webhook, for code

For anything automated, set webhook_url and webhook_events on the project. The events worth subscribing to for monitoring are run.finished (status, pages, counts, coverage, stop reason) and page.changed (each added, modified or removed page, with words added and removed). There are also page.blocked, site.failing / site.recovered and sitemap.added / sitemap.removed if you want to know when a site starts refusing you or restructures.

arc.projects.update(project["id"], config={
    "webhook_url": "https://hooks.yourapp.com/mesharc",
    "webhook_events": ["run.finished", "page.changed"],
})

Every delivery is a POST with a JSON body {id, event, createdAt, data} and the header X-MeshArc-Signature: sha256=<HMAC-SHA256 of the raw body>. Verify it against the raw bytes before parsing, and answer with a 2xx within 10 seconds:

import hashlib, hmac, json, os
from http.server import BaseHTTPRequestHandler, HTTPServer

SECRET = os.environ["MESHARC_WEBHOOK_SECRET"].encode()

class Hook(BaseHTTPRequestHandler):
    def do_POST(self):
        raw = self.rfile.read(int(self.headers["Content-Length"]))
        expected = "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
        if not hmac.compare_digest(expected, self.headers.get("X-MeshArc-Signature", "")):
            self.send_response(401); self.end_headers(); return

        self.send_response(204); self.end_headers()   # acknowledge first
        msg = json.loads(raw)
        if msg["event"] == "page.changed":
            queue_for_review(msg["data"])             # your code, off the request path

HTTPServer(("", 8080), Hook).serve_forever()

Acknowledge first and work afterwards. A missing answer, a 5xx, a 408 or a 429 is retried six times over about a day (1 m, 5 m, 30 m, 2 h, 6 h, 12 h) before the delivery is paused, so a deploy or a short outage on your side does not lose a change. Any other 4xx fails at once — useful when you return 401 to a forged request.

Stage 3: the table

Alerts are for today. For "what did this page say before the 14 March update", you want every version in a database. MeshArc pushes to 18 destinations on every plan, PostgreSQL among them, without glue code.

Add a Postgres connection once (its secret is the dsn), then a destination on the project. The settings, from the connectors and destinations docs:

The default columns already carry the change: change_class, added_words, removed_words and changed_fields, beside url, crawled_at, content_hash, markdown, title, canonical_tag and the rest. If you only care about part of a site, a shape filters rows (say, path matching /guidance/fees/*) and maps fields into typed columns; POST /destinations/preview returns the shaped rows of the last run before you save anything:

curl -X POST https://api.mesharc.dev/api/v1/destinations/preview \
  -H "Authorization: Bearer $MESHARC_API_KEY" \
  -d '{"project_id": "<id>", "columns": ["url","crawled_at","change_class","added_words","changed_fields"], "shape": {}}'

Once rows are landing, the questions are plain SQL:

-- What changed in the last 30 days, biggest edits first
SELECT url, change_class, added_words, removed_words, changed_fields, crawled_at
FROM site_changes
WHERE change_class IN ('added', 'modified', 'removed')
  AND crawled_at > now() - interval '30 days'
ORDER BY added_words + removed_words DESC;

-- Every version of one page
SELECT crawled_at, content_hash, left(markdown, 200)
FROM site_changes
WHERE url = 'https://www.example.gov/guidance/fees'
ORDER BY crawled_at;

One limit to know: a single sync writes at most 20,000 rows. For a monitor sending only changed pages that is rarely reached; for a full re-sync of a large site, set incremental: false deliberately and expect several syncs.

Where other tools are better

An honest map, checked on 28 October 2026:

• Firecrawl /monitor runs as often as every 5 minutes; MeshArc's fastest schedule is hourly. Its JSON change-tracking mode gives a per-field diff keyed by path (plans[0].price) with previous and current values, and it can watch web-search results for new hits, not just known pages. Its Slack integration is OAuth-based and set up in the dashboard. What it does not do is write to Postgres or a warehouse: changes leave via webhook, email or Slack, and the table is yours to build. Its webhook retries are three attempts at 1, 5 and 15 minutes. A fuller comparison is coming in its own post; the price side is in Firecrawl pricing explained.

• changedetection.io is open source and self-hostable. If you have a server and a short list of URLs, it is hard to argue with free.

• Visualping and Distill are built for people, not pipelines: visual diffs, browser extensions, email. For a marketing team watching ten competitor pages, they are friendlier than any API.

• Prices that move every few minutes belong to a live feed, not a crawler. Hourly is the floor here, by design.

Where MeshArc is the stronger fit: a whole section rather than a URL list, refusals that never masquerade as deletions, unchanged pages that cost nothing, and a change history that lands in your own Postgres, BigQuery or Snowflake without a line of ETL. The same pipeline, pointed at a vector store, is how we keep a RAG knowledge base fresh. Side-by-side figures are on compare, and the use case in full is on website change monitoring.

The short version

• Watch a section, not a URL list: a project with include_paths and a schedule, seeded from the sitemap.

• Noise is stripped before the diff; refused pages are withheld, and removals are held back on any run that reaches under 90% of the site.

• chat_url for Slack, Teams or Discord, with notify_min_words to keep it quiet; webhook_url with an HMAC signature for code.

• A Postgres destination keyed on url+content_hash gives one row per page version, with change_class and word counts ready to query.

• Change records, alerts, webhooks and destinations are free; unchanged pages carried forward by lastmod are free.

Point the playground at the section you keep checking by hand and see what a page returns, or start free with 1,000 credits and no card, set the baseline today and read the first diff tomorrow.