# MeshArc > A scraping pipeline for AI and data projects, with web search and an AI research agent. The ladder climbs only as far as a site makes it, every run is diffed against the last, and clean, structured, de-duplicated content lands wherever your data lives. MeshArc fetches a page the cheapest way that works -- plain HTTP first, a browser or a residential exit only when a site refuses -- and bills by what the page took. Every run is compared with the one before, so a project reports what changed, and results can be pushed to a database, a bucket or a vector store. It also searches the web and runs an AI research agent that answers a question with the pages it used -- for one question, every row of a table, or on a schedule. It is used through an HTTP API (scrape, crawl, map, extract, web search, the agent), Python and Node SDKs, an MCP server and a web app. ## Documentation - [Overview](https://mesharc.dev/docs): What MeshArc is and how it thinks: the ladder, the judge, what a page costs, change detection, web search and the agent, where the pages go. - [Getting started](https://mesharc.dev/docs/getting-started): Ten minutes from sign-up to a site being watched: the playground, a first project, webhooks and destinations, an API key, the SDKs, web search and the agent. - [Using the app](https://mesharc.dev/docs/app): Accounts and workspaces, roles, API keys, the playground and web search, agents and tables, every project screen, connectors, destinations, exports, monitor, billing. - [Configuration](https://mesharc.dev/docs/configuration): Every setting a project, a crawl, a scrape or the playground takes — its meaning, its default, what it costs. - [Connectors and destinations](https://mesharc.dev/docs/connectors-destinations): The providers, what each needs, how a destination writes rows, the shaping language. - [The API](https://mesharc.dev/docs/api): Authentication, conventions, every endpoint, webhooks, error codes, and what a page row says. - [SDKs and MCP](https://mesharc.dev/docs/sdks): The Python and Node SDKs and the MCP server for agents. - [LangChain](https://mesharc.dev/docs/langchain): The LangChain integration: a document loader for scrape, crawl, map and changes, web search and scrape tools for agents, and keeping a vector store current. - [Credits, plans and billing](https://mesharc.dev/docs/billing): What a page, a web search and an agent run cost, the plans and their limits, overage, payments. ## Product - [Pricing](https://mesharc.dev/pricing): The plans, their credits and limits, and what one page costs on each fetch mode. - [Developers](https://mesharc.dev/developers): The endpoints, the SDKs and how a request is made. - [MCP server](https://mesharc.dev/mcp): Web data for Claude, Cursor or any MCP client. - [Compare](https://mesharc.dev/compare): MeshArc beside other scraping and monitoring tools, feature by feature. - [Changelog](https://mesharc.dev/changelog): What changed in MeshArc, newest first. ## Use cases - [Add any website to your data pipeline](https://mesharc.dev/use-cases/data-pipeline): Add websites to your data pipeline. MeshArc crawls, de-duplicates and upserts web data into Postgres, BigQuery, Snowflake or S3 on a schedule. No ETL code. - [Clean web data for AI, ready to train on](https://mesharc.dev/use-cases/ai-training-data): Collect clean, de-duplicated web data for AI training, evaluation and fine-tuning. Markdown and JSON from any site, rebuilt as your sources change. - [A RAG pipeline that never serves stale answers](https://mesharc.dev/use-cases/rag-pipeline): Keep your RAG pipeline accurate. MeshArc re-embeds only the pages that changed, removes deleted pages from your vector store and exports llms.txt. - [Website change monitoring for whole sites, not single pages](https://mesharc.dev/use-cases/website-change-monitoring): Monitor website changes across entire sites and sections. Alerts only when content that matters changes, with confirmed removals and field-level diffs. - [Know the moment a competitor changes its price](https://mesharc.dev/use-cases/competitor-price-monitoring): Track competitor prices, plans and features automatically. MeshArc extracts prices as numbers and alerts you the moment they change. Free to start. - [The SEO website crawler that remembers last week](https://mesharc.dev/use-cases/seo-website-crawler): An SEO website crawler that remembers last week. Catch broken links, changed titles and canonicals, orphan pages and migration issues automatically. - [Your own Wayback Machine for the sites that matter](https://mesharc.dev/use-cases/website-archive): A private Wayback Machine alternative. Archive the sites you care about on your schedule, keep every version and export any snapshot as CSV or JSONL. ## Free tools - [Convert PDF to Markdown, free](https://mesharc.dev/tools/pdf-to-markdown): Convert PDF to clean Markdown online, free. Keeps headings, lists and tables, strips page furniture, and works on reports, manuals and whitepapers. - [Free llms.txt generator](https://mesharc.dev/tools/llms-txt-generator): Generate llms.txt and llms-full.txt for any website in seconds, so AI assistants read your site correctly. Free, from your own sitemaps. - [Convert any web page to Markdown](https://mesharc.dev/tools/html-to-markdown): Convert any web page or URL to clean Markdown. Strips navigation, ads and footers and keeps the main content. Free online tool with API access. - [Free broken link checker](https://mesharc.dev/tools/broken-link-checker): Find broken links and 404 pages for free. Checks every link on a page, internal and external, tells a dead link from a blocked one, and re-checks on a schedule. - [Free sitemap checker and URL extractor](https://mesharc.dev/tools/sitemap-checker): Check any website’s XML sitemap and extract every URL. Finds sitemap indexes, gzip, news, image and video sitemaps and lists URLs by section. Free and instant. ## Comparisons - [The Firecrawl alternative for teams that re-crawl the same sites](https://mesharc.dev/firecrawl-alternative): Looking for a Firecrawl alternative? MeshArc never bills blocked pages or 404s, reads protected sites for less and pushes data to 18 destinations. - [The Apify alternative for crawling the same sites on a schedule](https://mesharc.dev/apify-alternative): Looking for an Apify alternative? Compare Apify pricing and features with MeshArc: scheduled crawls, change tracking and 18 destinations, no actors to maintain. - [The Bright Data alternative for the sites you crawl again and again](https://mesharc.dev/bright-data-alternative): Comparing Bright Data competitors? See Web Unlocker pricing next to MeshArc’s: plain pages from $0.14 per 1,000, repeat visits to protected sites from $0.28. - [The ScrapingBee alternative for crawling whole sites, again and again](https://mesharc.dev/scrapingbee-alternative): Looking for a ScrapingBee alternative? MeshArc never bills a 404, re-crawls on a schedule, tracks what changed and sends data to 18 destinations. - [The ZenRows alternative for crawling protected sites on a schedule](https://mesharc.dev/zenrows-alternative): Comparing ZenRows pricing? MeshArc crawls whole sites, never bills a 404, reads protected sites from $0.28 per 1,000 after the first visit and tracks changes. - [The Visualping alternative for monitoring entire websites](https://mesharc.dev/visualping-alternative): A Visualping alternative for teams that monitor entire sites. Track sections through sitemaps, filter noise and send changes to Slack, email or your database. - [The Distill.io alternative for monitoring whole sections](https://mesharc.dev/distill-io-alternative): A Distill.io alternative for whole sites, not single pages: sitemap-aware checks, noise filtered out, changes sent to Slack, email or your database. - [The changedetection.io alternative for watching whole sites](https://mesharc.dev/changedetection-io-alternative): A changedetection.io alternative for whole sites: every page in a sitemap section checked each run, removals confirmed, changes sent to Slack or your database. - [The ChangeTower alternative for website change detection at site scale](https://mesharc.dev/changetower-alternative): Looking for a ChangeTower alternative? MeshArc detects website changes across entire sections, filters noise and sends field-level diffs where your team works. ## Search and agents - [A Google search API that hands back the pages, not just the links](https://mesharc.dev/web-search-api): A Google search API that returns organic results and, in the same call, each page as clean Markdown. Bing fallback, news search, refused searches free. - [An AI research agent that shows where its answers come from](https://mesharc.dev/ai-research-agent): Ask a question in plain English. The MeshArc AI agent searches the web, reads the pages and answers in your format, citing the pages it used. - [Upload a list. Get every row researched, with its sources.](https://mesharc.dev/ai-data-enrichment): Upload a CSV of companies or products. The MeshArc AI agent researches every row on the web and adds the answers as new columns, with their sources. - [Know the morning your Google ranking moves](https://mesharc.dev/rank-tracking-api): Track where any page ranks on Google for the searches you care about. Scheduled searches report new, dropped and moved results, with alerts and caps. ## Blog - [How to keep a RAG knowledge base fresh without re-embedding everything](https://mesharc.dev/blog/keep-rag-knowledge-base-fresh): A nightly rebuild keeps a RAG index current but leaves zombie chunks behind and rewrites thousands of identical vectors. Here is how to re-embed only what changed, with and without MeshArc. - [Firecrawl pricing explained: what 1,000 pages really cost](https://mesharc.dev/blog/firecrawl-pricing-explained): Firecrawl's headline is 1 credit per page. The real bill depends on your plan, your formats and how many of your URLs come back as 404s. Here it is in dollars. ## Optional - [The documentation as one text](https://mesharc.dev/llms-full.txt): Every docs chapter in a single file. - [Privacy policy](https://mesharc.dev/legal/privacy): What is collected, why, for how long, and your rights. - [Terms](https://mesharc.dev/legal/terms): The terms of service.