Guides
Scrape a Cloudflare-protected site for $0.40 per 1,000 pages: the session rung
30 Oct 2026 · 7 min read
Web scraping API and website crawler for AI and data teams
The web scraping API that remembers every site
Crawl any website, get clean Markdown or structured JSON, and know exactly what changed since the last run. MeshArc delivers the results straight to your database, warehouse or vector store — and you only pay for the pages it actually read.
Reach every page. Remember every site. One API call crawls a site; one more keeps it watched. A refused page, a 404 and a page that did not change all cost nothing.
no card · Python and Node SDKs · MCP server for Claude and Cursor · blocked pages and 404s are free
102 / 112
10 / 10
508,290
0
every number on this page came off a real run, recorded in the repo
§one platform instead of three tools
One platform instead of three tools.
Most teams glue together a scraper, a change monitor and an ETL job, then spend their weeks keeping all three alive. This is the three of them as one thing.
01
Read the sites that refuse
A ladder from a plain fetch to a real Chrome to a residential address to a paid answer. A refused page climbs one rung, never two; a refusal costs nothing; and the site remembers where it landed, so the platform learns a site once and every workspace benefits.
202
02
Know what changed, not what flickered
Every run is compared with the last, page by page and field by field. Consent banners, rotating widgets and volatile blocks are suppressed, so “Accept all cookies” never reads as fourteen modified pages. A page the site declined this time is withheld, never “removed”.
field-level
03
Land it where your data lives
Rows push themselves after each run to object stores, warehouses, databases, vector stores and streams — filtered, mapped and dropped on the way out, upserted by URL and content hash. Vector stores chunk and embed on your key.
18 destinations
new · web search and AI agents
Ask the web a question. Get the answer with its sources, then keep it watched.
Search Google and get the pages, not just the links. Hand a question to an AI agent that reads the web for you and shows the pages behind its answer — for one question, or every row of a spreadsheet. Then put it on a schedule and hear only when the answer changes.
01the ladder
A page costs what it took to read.
You never choose the mode — the ladder starts cheap and climbs one rung per refusal, never two. A page that needs more than your tier allows is not charged and is marked TIER_LIMIT: “this page costs more than you allow”, with the rung that would have read it. maxCredits caps any single page.
1 → 40
6 s
34 s
the ladder — cheapest rung first, one rung per refusalcredits / page
- 1Fetchhttp1
- 2Fetch with sessionhttp2
- 3Renderbrowser4
- 4Render, residentialstealth16
- 5Render, solvedstealth22
- 6Unlockedstealth40
Render, residential includes 1 MB; each started megabyte past the first is 4 credits.
every response gets a typed verdict, a confidence and an action
- okkeep
- thinkeep
- shellclimb → render
- blockedclimb
- rate-limitedwait
- captchasolve
- geo-blockedneed an exit
- login wallstop
02change detection
The diff is the product, not a flag.
Added, removed, modified, same, withheld, unverified, skipped — per page, per run. A page the site declined this time is withheld, never “removed”. Accept or ignore each change; mail a run’s record, or a digest on Monday morning in your own time zone.
run #48 vs #47 — 142 pages compared6 shown
- modified/news/2026/rail-funding-packagebody · 2 fields
- added/news/2026/coastal-defence-reviewnew in sitemap
- modified/news/2026/q3-procurement-noticepublished_at
- withheld/news/2026/minister-speech-cardiff403 · was ok
- same/news/2026/harbour-works-consultationbanner only
- same/news/indexrotating widget
/news/2026/rail-funding-packageword diff
h1
Rail funding package — consultation decision
noise suppressed — not a change
Accept all cookiesRelated: 3 rotating linksLast reviewed 16 Sep
- titleRail funding package — consultation→Rail funding package — decision
- published_at2026-09-09T08:00Z→2026-09-16T06:30Z
- authorPress office→Press office
- priceselector break
03sitemaps & discovery
The sitemap is kept, diffed and selectable by section.
robots.txt, well-known paths, <link rel=sitemap> and feeds, then indexes walked two deep — gzip, text sitemaps, lastmod, Google News, hreflang, image and video sitemaps. Every file is fetched through the ladder, so a blocked sitemap is climbed, not skipped.
508,290
500k
lastmod
sitemap index · 178 files · sections, not files+412−9since last snapshot
- /news/34 files128,440+412 −9
- /guidance/61 files204,118+38
- /publications/52 files141,902+1,204
- /government/people/9 files21,650—
- /help/4 files12,180−2
selecting a section writes the include / exclude globs — /news/**, /guidance/2026/**
link graph, per run
- orphans38
- unlisted12
- dead ends4
- coverageok
04destinations
Eighteen places to land, and none of them cost extra.
Push, not pull. Every connector is tested from the UI before it runs, gets its own sync history, and can be scoped to one project or the whole workspace. Vector stores chunk and embed on your embedding key.
Object stores5
- Amazon S3
- S3-compatible
- Google Cloud Storage
- Azure Blob
- Local disk
Warehouses5
- BigQuery
- Snowflake
- Redshift
- Databricks
- ClickHouse
Databases3
- PostgreSQL
- MySQL
- MongoDB
Vector stores4
- Pinecone
- Qdrant
- Weaviate
- pgvector
Streams2
- Kafka
- HTTP webhook
shape on the way out — no codedestination: PostgreSQL
- filterurl glob /news/*
- mapprice → number → ×100 → int
- mappublished_at → date
- droprawHtml
upsert by url + content hash · the same run lands raw in S3 and shaped in PostgreSQL
exports, if you would rather pull
- pages
- markdown
- changes
- fields
- sitemap
- llms.txt
- llms-full.txt
CSV, JSONL or a text file for a model · per run or per project · POST /projects/{id}/export
05what comes back
A full-featured web scraper, in any format you need.
This is a complete scraping API, not only a monitor. Point it at one URL or a whole site and choose what comes back — then narrow it until you are paying for nothing else.
every page, in the shape you asked for
and only what you asked for
06for developers
One job envelope, a committed OpenAPI spec, and an MCP server.
Every asynchronous verb returns the same envelope — id, kind, status, progress, creditsUsed, data, next — with cursor pagination, Idempotency-Key, X-Request-Id and X-MeshArc-Credits on every response, and typed error codes. Keys carry scopes, a project allow-list, an expiry and their own rate limit.
POST /crawlcurl
curl -X POST https://api.mesharc.dev/v1/crawl \
-H "Authorization: Bearer $MESHARC_KEY" \
-H "Idempotency-Key: rail-funding-48" \
-d '{
"url": "https://www.gov.uk/news",
"mode": "sitemap_first",
"formats": ["markdown", "json"],
"fields": ["title", "published_at", "author"],
"tier": "browser"
}'
pip install mesharcpython
from mesharc import MeshArc
arc = MeshArc() # reads MESHARC_API_KEY
job = arc.crawl("https://www.gov.uk/news", useSitemap=True, wait=True)
for page in job.pages():
print(page["url"], page["verdict"], page["credits"])
# keep it, and it watches
project = job.keep(name="gov.uk news", schedule="daily")
for entry in arc.changes(project["id"])["change"]["feed"]:
print(entry["kind"], entry["url"])
MCP server — the tools an agent gets15 tools
- scrape_urls
- map_site
- search_web
- crawl_site
- create_project
- get_project
- update_project
- delete_project
- list_projects
- start_run
- list_runs
- list_pages
- get_changes
- get_job
- cancel_job
formats per page
- markdown
- text
- rawHtml
- cleanHtml
- links
- raw
- screenshot
- json
PDF and Word (.docx) files parsed to text and markdown · structured fields from what the page already declares
07pricing
Priced by the page it took, not the page you hoped for.
A plain fetch is one credit, a real browser four, a residential address sixteen. Change detection, sitemaps, destinations, exports and webhooks are on every plan. Refused pages and 404s are free.
- Free$01,000
- Starter$29/mo100,000
- Growth$99/mo500,000
- Scale$349/mo2,500,000
- Enterprisefrom $1,500/mono ceiling
- A page costs what it took to read.
- You pay for pages, not for walls.
- A site you have read before starts where it last answered.
- Change detection, sitemaps and destinations are included.
- Extraction runs on your own model key.
08built for
The teams that depend on web data.
Each of these is a page of its own: the problem in that team's words, the configuration that answers it, and what comes back.
AI and RAG teamsKeep a knowledge base freshData engineersA website as a tableProduct and foundersA competitor’s price, the morning it movesCompliance and legalRegulators, watched by the sectionSEO and agenciesEvery changed title and canonicalAgent buildersThe web, in one installSales and RevOpsA list of accounts, researchedStrategy and researchA cited answer, not a pile of tabsGrowth and SEOWhere you rank, every morning
09before you sign up
The questions a careful buyer asks.
What is a web scraping API?
A web scraping API fetches web pages for you and returns their content in a clean, usable form, handling proxies, JavaScript rendering and anti-bot measures out of sight. This one adds two things most do not: it remembers every site, so it can say what changed since last time, and it delivers the results into your own data stores.
How is this different from other web scraping tools?
Most scraping APIs sell requests: you are billed for the attempt, whatever came back. This one sells results — a refused page, a 404 and a page that did not change are all zero — and the second crawl of a site costs less than the first, because the rung that worked is remembered.
Do I have to choose a fetch mode?
No. Every page starts as a plain fetch and the ladder climbs one rung per refusal — a session, a real browser, a residential address, a solved widget, a rented unblocker. You set the ceiling per project and maxCredits per request; the response says which rung read each page and what it cost.
What does a blocked page cost?
Nothing. A page the site refused, whatever the ladder spent finding out, and a 404 or 410 are on the ledger at zero. A page that needs more than your tier allows is not charged and comes back TIER_LIMIT with the rung that would have read it.
Does it solve captchas?
Cloudflare’s “Just a moment” page needs no solver — a real browser waits it out and earns the site’s session, so every page after it is read at 2 credits. Captcha widgets on forms (Turnstile, reCAPTCHA v2 and v3) are opt-in per project: a checkbox is clicked for free first and an answer is bought only when the pictures come. hCaptcha is detected and refused.
Where does my data go?
Into your own stores. After each run, rows push to S3, GCS, Azure Blob, BigQuery, Snowflake, Redshift, Databricks, ClickHouse, Postgres, MySQL, Mongo, Pinecone, Qdrant, Weaviate, pgvector, Kafka or a webhook — filtered, mapped and upserted by URL. Or pull CSV and JSONL exports. Model extraction and embeddings run on your own keys. Read about the eighteen destinations.
How is this different from Firecrawl, ScrapingBee or ZenRows?
A cheapest-first ladder and success-only billing are not ours alone. What is: the session rung that reads a walled site at 2 credits after one render, a memory of every site shared across workspaces, and change detection, sitemaps and destinations sold as the product rather than bolted on. Prices are compared in dollars per thousand pages. Read about the comparison.
Can it search the web?
Yes. A search reads Google’s organic results — Bing’s when Google refuses — and can return each result’s page as Markdown in the same call. A results page every engine refused is free, and the same search within the hour costs nothing. Read about the web search API.
What does the AI agent do, and what does it cost?
Ask it a question and it searches the web, reads the pages it needs and answers — in the format you ask for, with its values linked to the pages they were read on. It pays for the pages it reads, like any request, and the model’s tokens at the provider’s price plus 20%, never past the cap you set on the run. Read about the research agent.
Can I run it over a spreadsheet?
Yes. Upload a CSV, write the question once with the columns in it, and every row is researched on its own, with a credit cap per row and the cost estimated before anything runs. The answers come back as new columns, linked to the pages they were read on. Read about data enrichment.
from the blog
What we learn running a crawler.
Guides
How to keep a RAG knowledge base fresh without re-embedding everything
24 Oct 2026 · 8 min read
Turn any website into data you can trust.
1,000 free credits, browser rendering included, no card. The playground answers in seconds.