# MeshArc documentation

> A scraping pipeline for AI and data projects, with web search and an AI research agent. The ladder climbs only as far as a site makes it, every run is diffed against the last, and clean, structured, de-duplicated content lands wherever your data lives.

---

# Overview

Source: https://mesharc.dev/docs

# MeshArc documentation

*Reach every page. Remember every site.*

MeshArc is a web scraping API and website crawler that turns URLs into clean Markdown, text, HTML or structured JSON, remembers every site between runs, tells you what changed, and delivers the results to your own data stores. It also searches the web and runs an AI research agent that answers a question with its sources — for one question, every row of a table, or on a schedule. It is a hosted service: a web app at `https://mesharc.dev`, an API at `https://api.mesharc.dev`, SDKs for Python and Node, and an MCP server for agents.

Everything about the product is here: how it works, how to get started, every screen, every setting with its default, the connectors and destinations, the API, the SDKs, and what it costs. Current as of 18 September 2026.

| | What it covers |
|---|---|
| [Getting started](/docs/getting-started) | Sign up, read a page, watch a site, get the pages somewhere, use the API |
| [Using the app](/docs/app) | Sign-up, workspaces and members, keys, the playground and web search, agents and tables, projects and every project screen, exports, monitor, billing |
| [Configuration](/docs/configuration) | Every setting a project, a crawl, a scrape or the playground takes — what it means, its default, what it costs |
| [Connectors and destinations](/docs/connectors-destinations) | The providers, what each needs, how a destination writes rows, the shaping language |
| [The API](/docs/api) | Authentication, conventions, every endpoint, webhooks, error codes, verdicts |
| [SDKs and MCP](/docs/sdks) | `pip install mesharc`, `npm install mesharc`, the MCP server |
| [LangChain](/docs/langchain) | `pip install langchain-mesharc`: a document loader, search and scrape tools, keeping a vector store current |
| [Credits, plans and billing](/docs/billing) | What a page costs, the plans, limits, overage, payments |

## How MeshArc thinks

**A project** is a site (a seed URL) plus its configuration: scope, sources, schedule, formats, browser steps, tier. It lives in a workspace. **A run** is one crawl of a project: pages, a stop reason, counts, a credit tally, a sitemap snapshot, a link graph, and a change record against the previous run. **A page** is one URL in one run: bodies, head metadata, structured fields, a verdict, the engine that read it, the tier, the cost.

**The ladder.** Every page starts on the cheapest engine that could read it and climbs only when refused. Eight engines, cheapest first:

| Engine (wire name) | What it is | Tier | Credits |
|---|---|---|---|
| `crawler`, `modern-ua`, `tls` | Plain HTTP with a real browser's TLS fingerprint | http | 1 |
| `minted` (http-jar) | The same request carrying cookies a browser earned | http | 2 |
| `browser` | A real Chrome, headed, persistent profile | browser | 4 |
| `browser-residential` | Chrome through a residential exit | stealth | 16, +4 per MB through the exit past the first |
| `solver` | A render plus a paid captcha token | stealth | 22 (a widget that passed on a click is billed as a render) |
| `unblocker` | A rented fetch: someone else's pool and solvers | stealth | 40 |

A refused page climbs one rung; the *site's* starting rung moves only when three of the last five pages needed it. What a host needs is remembered per domain, across every workspace, for a day — so the platform learns a site once.

**The judge.** Every response gets a typed verdict: `ok`, `thin` (a short page that is still a page — kept, with `warnings: ["short"]`), `shell` (an app shell whose content is script — rendered once), `blocked_status` / `blocked_challenge` (climb), `rate_limited` (wait), `captcha` (solve), `geo_block` (an exit elsewhere), `login_wall` (stop), `missing` (a 404/410, or a 200 that is the site's not-found page — stop). A short page whose markup repeats is what the markup says — `shape: listing / table / form` — and is `ok`.

**What you pay.** A page costs the rung that answered it. A page the site refused, a 404, a soft 404, a page robots.txt kept us from, a cached copy: 0. A render bought to check a short page that turns out unchanged is billed at the http rung. A map costs one credit per sitemap file read. Every job envelope carries `creditsUsed`; every response carries `X-MeshArc-Credits`.

**Change detection is the product.** Two runs of a project give a change record: pages added, removed, modified, same, withheld, unverified, skipped; field changes (title, description, canonical, robots, h1, Open Graph, Twitter card, lang); "the selector broke" signals; volatile blocks filtered. A run that reached under 90 % of the site cannot tell "removed" from "not reached", so removals are withheld and the record says so.

**Search and the agent.** A web search reads Google's results page — Bing's when Google refuses — and returns the organic results, optionally with every result's page; it is priced like a page. The agent takes a question, searches, reads the pages it needs through the same ladder, and answers as text or in the fields you ask for, with the pages it used; it pays for what it reads and its model's tokens, within a cap you set. Both can be kept on a schedule as **monitors** that report what changed, and a **trigger** can start one when a page of a project changes. [The API](/docs/api) has the routes; [Using the app](/docs/app#agents) the screens.

**Where the pages go.** Webhooks (signed), 19 connectors (object stores, databases, warehouses, vector indexes, streams) bound as destinations with a declarative shape, and exports (JSONL / CSV) of pages, markdown, changes, fields and the sitemap.

---

# Getting started

Source: https://mesharc.dev/docs/getting-started

# Getting started

Sign-up to a site being watched takes about ten minutes. There is nothing to install for the app; the SDKs are one line each.

## 1. Create your workspace

Go to **mesharc.dev → Sign up** with a work email and a password. The first account is the workspace's owner. A six-digit code arrives by mail within a minute; enter it and the workspace is yours. Until it is entered you can look around, but nothing that spends credits will start. Once you are in, turn on [two-factor authentication](/docs/app#two-factor-authentication) under Profile — it takes a minute with any authenticator app.

The free tier gives **1,000 credits, once** — enough to read a few hundred plain pages or a few dozen rendered ones — with the browser tier, two projects and 500 pages per run. Nothing renews and nothing is charged; when you want more, the [plans](/docs/billing#plans) start at $29.

## 2. Read one page in the playground

**Playground → Single page**, paste a URL, press **Extract**. In a few seconds you have the page as markdown, and the **Request** tab tells you what happened: the rung that won (a plain fetch, a render, a residential exit), the engine, the status and the words. A page that could not be read says why, with the response's evidence. Billing's ledger shows what each request cost. Try a JavaScript-only page and watch the ladder climb to a render on its own; try a page that refuses and see it cost 0.

The rail on the left is every setting a project has — the same ones [Configuration](/docs/configuration) goes through one by one. Change one and run again. The snippet under the result is the exact API request that just ran, in curl, Python or Node.

**Playground → Crawl site** does the same for a whole site with a page budget and a depth; **Keep** turns the result into a project. **Playground → Map URLs** lists what a site's sitemap declares before you fetch any of it.

## 3. Watch a site

**Projects → New project**: a seed (a domain or a URL), an optional name, a page budget. The project runs **weekly** and the form says what that costs at your budget. **Run now** starts the first run; the second run, a week later or whenever you press Run again, produces the first change record.

While it runs, **Runs** shows the log live. Afterwards:

- **Pages** — every page with its verdict, engine, credits and how the crawl found it.
- **Sources** — what the sitemap declared, what the links found, and which sections to keep.
- **Configure** — the settings, grouped, with what each one does and charges; a JSON pane; a dry run.
- **Changes** — after the second run: pages added, removed and modified, field changes, word-level diffs.

[Using the app](/docs/app) walks through every screen.

## 4. Get the pages somewhere

Three ways, all included:

| | When to use it | Where |
|---|---|---|
| **Webhook** | Your system should hear when a run finishes or a page changes | Configure → Advanced: a URL, the events, a signing secret; deliveries listed with a resend |
| **Destination** | The pages should land in your database, warehouse, bucket, vector index or stream after every run | Connectors (the credentials, tested once) → Destinations (the target, the columns, a shape) |
| **Export** | You want a file now | Exports, or a selection on the Pages screen: JSONL or CSV |

## 5. Use the API

**Settings → API keys → New key.** The key is shown once. Then:

```bash
curl https://api.mesharc.dev/api/v1/scrape \
  -H "Authorization: Bearer mesharc_..." -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/pricing", "formats": "markdown,links"}'
```

```bash
pip install mesharc        # Python 3.10+
npm install mesharc        # Node 20+
```

```python
from mesharc import MeshArc
arc = MeshArc("mesharc_...")
page = arc.scrape("https://example.com/pricing")
print(page["markdown"], page["credits"])
```

Every call answers with what it cost (`X-MeshArc-Credits`, `creditsUsed`) and a request id to quote to support. The full [API reference](/docs/api) covers every route; the [SDKs and the MCP server](/docs/sdks) wrap them for Python, Node and agents.

## 6. Invite the team

**Settings → Members → Invite**: an address and a role. Viewers read; members run things; admins manage keys, connections and people. [Members and roles](/docs/app#accounts-and-workspaces) spells out who can do what.

## 7. Search the web and ask the agent

**Playground → Search**: a query, a country and language if you like, then **Search**. You get Google's organic results — Bing's when Google refuses — as data; tick **Scrape each result to markdown** and every result's page comes back too. **Save and schedule** keeps the search as a monitor that tells you what is new, what dropped and what moved.

**Agents → + New**: write what you want found, naming the company, site or topic, and **Send** — or pick an example to watch one work. The agent searches, reads the pages it needs and answers as text or in the columns you describe, with its sources; ask a follow-up in the same conversation. **Run on a table** asks one question of every row of a CSV, with a credit cap per row. A run stops at its cap (2,000 credits unless you set another). [Credits, plans and billing](/docs/billing) says what a search and an agent run cost, and [Using the app](/docs/app#agents) walks through the screens.

## What to expect

- **A page costs what it took to read.** Most pages are 1 credit. A render is 4. A page the site refuses is 0, and so is a 404. You never pick the engine; you set a ceiling.
- **Politeness is real.** robots.txt is obeyed, including its crawl-delay; one request at a time per host by default; a site that starts refusing is slowed down, not hammered. The one exception is the search engine's results page a web search reads; the pages it goes on to read obey robots.txt.
- **Your playground never waits behind your own crawl.** Interactive work has its own lane.
- **A site you have read before starts where it last answered** — the platform remembers what a host needs.
- **Nothing is read on credit.** When a workspace runs out, runs stop and say so.

---

# Using the app

Source: https://mesharc.dev/docs/app

# Using the app

The app is at `https://mesharc.dev`. Everything it shows comes from the [API](/docs/api); nothing is computed in the browser that the API does not also return.

## Accounts and workspaces

- **Sign up** (`/signup`) with a work email and a password. The first account of a workspace is its **owner**. A six-digit **verification code** arrives by mail (30-minute lifetime); until it is entered you can look around, but creating a project, starting a run, issuing a key or inviting someone answers `403 email_unverified`. The shell shows a banner with a link to enter the code.
- **Sign in** (`/login`). Sessions last 30 days and end after 14 idle days; "Keep me signed in" is the default. `Profile → Sessions` lists them and ends any. An account with [two-factor authentication](#two-factor-authentication) on is asked for a code after the password.
- **Forgot password** (`/forgot`) mails a reset link; `Profile` changes it while signed in.
- **Google or GitHub.** Where the deployment offers them, *Continue with Google* and *Continue with GitHub* sit above both forms. The first time, a MeshArc account is made with the provider's verified email address — verified already, so no code is mailed — and its own workspace, or the invited one. If an account with that address already exists it is linked to it, and a mail says so; for a Google account that is not a Gmail address or managed by a company (Google Workspace), link it from Profile instead. An account with two-factor on is still asked for its code. `Profile → Sign-in methods` lists the linked accounts, links another and removes one — never the last way in. An account made this way has no password until one is set through *Forgot password*.
- **Workspaces.** An account can belong to several; the sidebar switches. `Settings → Workspace` renames it.
- **Members and roles** (`Settings → Members`): `owner` > `admin` > `member` > `viewer`. A **viewer** reads everything. A **member** creates and configures projects, starts runs, scrapes, crawls and maps, reviews changes, tests webhooks and syncs destinations. An **admin** also issues and revokes API keys, sets the rate ceiling, deletes projects, rotates webhook secrets, manages connections and destinations, invites and removes members, sets roles, renames the workspace and opens the billing portal. The **owner** is the admin who cannot be removed. Invites go by mail with a link that lives seven days; an invite names the role.
- **Profile**: name, time zone, two-factor authentication, whether you receive **run emails** (a run record when a project finishes) and the **weekly digest** (Monday from 08:00 in your time zone: every project's week in one message).
- **Notifications on screen.** On a server with no mail host set up (a self-hosted or development install), run records and monitor alerts are not emailed: they appear as popups in the bottom corner of the app instead, newest first, up to a week old. *Open* goes to the run's changes or the monitor; × dismisses one. Where a form offers email, it says when this is how it will arrive.

### Two-factor authentication

A second factor is a six-digit code from an authenticator app — Google Authenticator, 1Password, Authy, Microsoft Authenticator, anything that does time-based codes — asked for after the password every time you sign in.

- **Turning it on** (`Profile → Two-factor authentication → Turn on`): enter your password, scan the QR code with the app (or type the key it shows), then enter the code the app displays. Every other session on the account is signed out, since none of them showed a factor. You are then handed **ten recovery codes**; keep them where you keep passwords, because they are shown only once.
- **Recovery codes** sign you in when the phone is lost. Each works once; the sign-in screen has a "Lost the phone?" link that takes one instead of an app code. After a recovery sign-in the Profile shows how many remain, and `New recovery codes` makes a fresh set of ten (against a current app code) — the old set stops working.
- **Confirming it's you.** With two-factor on, a few actions ask for the code again if the session last showed it more than ten minutes ago: changing the password, turning two-factor off, making new recovery codes, creating an API key, changing a member's role or removing them, and changing the workspace's two-factor rule. A small dialog asks for the code and the action then goes through.
- **Turning it off** takes your password *and* a current code. A mail goes to the account's address whenever two-factor is turned on or off, when a recovery code is used, and when new codes are made.
- **Without a password** (an account that signs in with Google or GitHub), turning two-factor on or off asks for a sign-in from the last ten minutes in place of the password; sign in again if it has been longer.
- **Resetting a forgotten password** on an account with two-factor also asks for an app code or a recovery code: an inbox alone is not enough to take over the account.
- **Requiring it for a workspace** (`Settings → Workspace → Require two-factor authentication`): an owner — who has it on for their own account — can make it a condition of acting. A member without it can still sign in and look around, but creating a project, starting a run, issuing a key or changing anything answers `403 mfa_required` until it is set up; the shell shows a banner with the link, and the members list marks who has it on.
- API keys are unaffected by any of this: a key was made by someone who had already passed the gate, and a program cannot read a phone.

## API keys

A key is shown **once** when it is created. It carries **scopes** (`read` < `write` < `admin`, matching the member roles), optionally a **set of projects** it may see (any other project — or another workspace's — answers 404, never 403), an **expiry**, and its own **rate limit** (`rpm`), reported on every response as `X-RateLimit-Limit / -Remaining / -Reset`. Keys start with `mesharc_`. Revoking is immediate.

The workspace's own ceiling is `max_rpm` (`Settings → Rate ceiling`, `PATCH /me`); concurrency scales with it so the rate to a host never exceeds it.

## The playground

One URL through the ladder, with every setting on the left as a request field. Four modes:

| Mode | What happens | API |
|---|---|---|
| **Single page** | One page, synchronously; the tabs (Markdown, Text, Clean HTML, Fields, Screenshot, Links, Images) follow the formats you chose, and **Request** says the rung that won, the engine, the status, the words and the content hash. A page that could not be read says why, with the response's evidence. What it cost is on Billing's ledger | `POST /scrape` |
| **Crawl site** | A one-shot crawl of the site with the page budget and depth from the rail; poll its pages; **Keep** promotes it to a project | `POST /crawl`, `POST /crawl/{id}/keep` |
| **Map URLs** | The site's URLs from its sitemap tree | `POST /map` |
| **Search** | Web or news results for a query: Google's organic results, Bing's when Google refuses, with country, language, freshness, the sites to keep to or leave out, and the results page (1–10). **Scrape each result to markdown** reads every result's page too. The header says how many results, what it cost and which engine answered; **Save and schedule** keeps it as a monitor; **All searches** lists the past ones | `POST /search`, `GET /search` |

The rail's settings are the same 84 controls as a project's Configure screen — see [Configuration](/docs/configuration) for each one; the request sends only what differs from the defaults, and the code snippet (curl, Python, Node) is a copy of the request that just ran.

## Agents

**Agents** is a chat. On the left, the workspace's **conversations**, the latest first, grouped by day (Today, Yesterday, Earlier this week, Older), each with its messages, credits and when — or *working* while a message runs — and a filter; under them the **Tables** run row by row. **+ New** opens a new conversation: write what you want found, or pick an example. **Start from**, under the message box, takes the sites it should read first (optional; without one it searches the web), and a site your message names is offered as **Start from …** in one click. The agent is only here: the Playground's old agent links open a new conversation with their example filled in.

In a conversation, each message is one agent run with its own status, budget, credits, steps and **Cancel** — billing is per message, as for any run. While a message works, its steps appear as it takes them (searches, pages read, maps, each with its cost), and the line under them says what it is doing now — "Working out what to do next…", "Reading example.com/pricing…"; the model's share closes the list ("2 model steps · 2,060 tokens", and its credits). When it answers, they fold into one line ("5 steps · searched twice · read 4 pages"), opened with a click. The answer shows its sources numbered, a link in the text cites its source's number, and **Copy**, **Download**, **Full trace**, **Keep and re-run** (not on a follow-up or a continuation) and — at a credit stop — **Continue with a new budget** sit under it. Proposals the message made appear under it with **Approve** and **Dismiss**.

**Ask a follow-up** in the box at the bottom: it remembers everything asked and found in the conversation, and pages already read in it are read again free (the message says how many it could reuse). The chips under the box — *Start from*, *Answer*, *Budget*, *Steps*, *Sites*, and *Model*, *Proposals*, *Logins* and *Data* when set — are the next message's options, kept from the last message until you change them; one changed from the default is marked, and a chip opens them all. A conversation of three messages or more opens with all but the latest folded under one line naming them ("↑ 2 earlier messages: …"); a click shows them, and what you ask after stays in view. You can send once the last answer has finished, or cancel it. **API request** shows the next message as `POST /agent/{id}/follow-up`. A conversation holds up to 30 messages and is kept 7 days after its last one.

**Run on a table** asks one question of every row of a table (`POST /agent/batch`):

1. **The table** — upload a `.csv` or paste one; its first line names the columns, and the first rows are shown.
2. **What each row asks** — the prompt, with a column put in as `{{column}}` (click a column to insert it). *Start each row at* picks the column holding each row's URL; with **Keep each row on its own site** ticked, each row reads only its own site. Without it, a warning says the rows may read any site.
3. **The answer's columns** — the fields each row's answer adds, or free text.
4. **What it may spend** — the most credits a row, and how many rows run at once. Under **Advanced**, the model, the steps a row and the **data providers** every row may look things up with.

**Check the table** asks the server what would run: how many rows, which cannot run and why, the first prompts as their rows will ask them, and about what it will cost against your balance. **Run** starts the rows that can. The batch's page fills in as rows end — each row's values, its status and its answer as new columns, with the page each value was read on and a link to the row's own run — and downloads as CSV or JSON lines. **Cancel** stops the rows not yet finished.

## Projects

The workspace table: every project with its pages, reach, blocked share, pending changes, schedule and last run. Select several and **Run**. **New project** takes a seed (a domain or a URL), an optional name and a page budget; a new project runs **weekly** and the form says what that costs ("about 400 credits a month at 1 a page"). **New from a list** creates many from seeds with a template project's config.

Each project has these screens.

### Overview

Stat cards (pages tracked, pending review, last-run coverage, credits), "what changed in the last run" with field-change chips and the top changed pages, a health panel (healthy / failing / recovered from the last two runs), and recent runs. **Run now** starts a run.

### Sources
Where the project's URLs come from:

- **The seed** and its links.
- **The sitemap tree** — discovered from robots.txt, the well-known paths and the seed page's link tags, indexes walked to their children; or only the files you list (`sitemap_mode: listed`); or off. Every file with its section label, URL count and `lastmod` range; a diff against the last snapshot (files and URLs added, removed, updated). Pick the **sections** a crawl keeps (`sitemap_include` / `sitemap_exclude` globs on the file URL or section) and see how many URLs the selection keeps before a run (**Preview**).
- **URL list** — up to 5,000 exact URLs, fetched as given, never filtered by the path globs.
- **Feeds** — up to 20 RSS/Atom feeds, read each run; every entry is queued first.
- **Patterns** — templates expanded into URLs: `/products/{1..500}`, `/docs/{intro,setup,faq}`; capped at 5,000 expansions.

The scope preview shows pages per depth with the excluded and beyond-budget bands, and an estimate (`~340 pages · ~1.4M tokens · ~4 min`).

### Configure
The whole configuration as 84 controls in seven groups — Output formats, Content filters, Crawl scope, Rows, Images & media, Fetch behaviour, Advanced — each with a tip that says what the engine does and what it charges; a JSON pane that round-trips to the controls with line-level parse errors; **Dry run** (resolve sources and sample URLs without spending); a staged-change diff; **Apply to…** (copy the saved config onto other projects by group); **Webhook deliveries** (every message sent, attempt by attempt, with **resend**); the signing secret and a **Send test**; a **chat channel** (a Slack, Discord, Teams, Mattermost or Google Chat incoming-webhook URL — the run record posts there when a run finds something, never when it compared clean; the URL is shown masked once saved); the **change digest** (a model and a focus line); and the danger zone (delete the project). [Configuration](/docs/configuration) has every control with its meaning and its default.

### Pages
Every page of a run with its verdict, engine, tier, credits, words, status and how it was found (seed / sitemap / link / URL list / feed / pattern). Filter with the query language (`status:403 depth:>2 flag:orphan`). Two **search** modes scan the run on the server: *content* (words and "phrases" over the markdown) and *selector* (a CSS selector or XPath over the stored HTML). A selection can be **re-crawled now** (a scoped run), **exported** (JSONL / CSV) or **excluded** (added to `exclude_paths`). **Removed** lists pages that were in an earlier run and not this one.

### Page detail
Five tabs: the rendered markdown, the sandboxed HTML, head fields (title, description, canonical, robots, h1, Open Graph, Twitter card, JSON-LD), structured fields with what resolved each one, and versions across runs with the dual-hash diagnostic (content hash vs HTML hash: a page whose HTML changed but whose content did not is a template edit, not a content change). A screenshot when the project takes one.

### Changes
The change record of a run against the one before:

- **Field changes** — title, description, canonical, robots, h1, Open Graph, Twitter card, lang — with the before and after, ranked.
- **Content diff** per page — word-level, side by side or unified, a volatile-block filter (dates, counters, "3 comments"), a section rail.
- **Markup comparison** — a noise filter, the DOM tree, selector-breakage repointing ("`.price` matched 40 pages last run and 0 now; the same nodes now match `.product-price`"), ignore rules.
- The **coverage banner**: a run that reached under 90 % of the site withholds removals, and the withheld URLs are listed under it.
- **Reviewed** marks per change.
- The **digest**, when the project has one (Configure → Advanced → *Change digest*, a model from Connectors → Models and a line on what you care about): a paragraph on what changed and why it matters, up to five points, and whether any of it is significant — written by the model from the changed paragraphs, the added pages' openings, the removed paths and the field diffs, on your own key. The same paragraph opens the run email and the chat message.

### Rows
For a project that reads its pages as records (Configure → **Rows**): every record its pages hold, what moved on the last run, and the records a run stopped finding. A record opens its own history — when it was first and last seen, and every column that ever moved, with the run that moved it. A board of roles, a table of recalls, a schedule of charges and a feed of posts are all lists of records, and watching them as pages only ever says "something changed". Exported as the **rows** and **row-events** datasets, and carried by the `row.changed` webhook.

### Runs
History with status, trigger (manual / scheduled / API / recrawl), pages, credits, stop reason and duration; the live log with time and level while a run is on (polled every 2 s); the engine breakdown; grouped errors; **Cancel**.

### Destinations
The project's destinations and the shared ones that include it, each with its target, mode, columns, shape and sync history ("synced 4 h ago · 84 written, 1,190 skipped"). **Sync now** writes the last run by hand. How a destination is set up, and how rows are shaped, is under [Connectors and destinations](/docs/connectors-destinations).

## Connectors

Workspace-level connections: a provider plus the credentials to reach it, named, and **tested** before it is trusted. The catalogue (21 providers) with a connect wizard per provider, then per-connection overview, mapping and settings. Secrets are never returned by the API and are encrypted at rest. The **Models** kind (any OpenAI-compatible or Anthropic chat endpoint on your key) is what a project's LLM enrichment stage uses. The **Data providers** kind — People Data Labs (company data) and Financial Modeling Prep (financial data), each on your own key — is what the agent looks things up with: tick one under **Advanced → Data providers** in the agent's options or a table's. A lookup costs no MeshArc credits (the provider bills your key), appears in the message's steps, and its record is one of the answer's sources. Neither a model nor a data provider can be a destination.

## Shared destinations

One target for many projects: one target that every project (**scope all**) or a chosen set (**scope projects**) writes after each of their runs, with `site`, `project_id` and `project_name` as columns so one table holds many sites. A shape (filter / map / drop) previewed against the last run before saving.

## Exports

Datasets streamed on request as JSONL or CSV: **pages**, **markdown**, **changes**, **fields**, **sitemap** — per run or per project, or restricted to a selection from the Pages screen. Two text files for a model: **llms.txt** (the usable pages as links with descriptions, by section — what an assistant reads to know what the site holds) and **llms-full.txt** (every usable page's markdown in one file, under its title and source URL — the corpus). Also every project's **webhook** and **chat channel** with its health from recent deliveries (healthy / failing / paused / idle), and every **destination** with its last sync.

## Monitor

**Proposed by agents**, at the top when there are any: what agent runs proposed to watch — a site to crawl, a search or a question to keep on a schedule — each with how often, why the agent proposes it, about what it would cost a month and the run that proposed it. **Approve** makes it — a monitor's first run starts at once, a project crawls at its first scheduled time or when you run it; **Dismiss** deletes it. Nothing runs or is charged before that. An agent run proposes only when **May propose things to watch** is ticked in the message's options; its own page lists what it proposed, decided or not, with a link to what was made.

**Scheduled**, under it: everything that runs on a schedule — crawl projects with a schedule, kept searches and kept agent runs — one row each, filtered by kind: what its last run found, when the next one is (or *paused*), what it has spent this month against its cap and about what a month costs, and who is told. In place: **Run now**, **Pause** / **Resume** (the schedule is kept; resumed, a slot that passed runs once), **Edit** (a project's Configure, a monitor's page), **History**, and **Delete**, which asks for the name to be typed.

Under it, **Triggers**: every trigger in the workspace, what it watches and does, and its last fire. A project's triggers are added on its Configure screen, a monitor's on its page: when a run changes something that matches — pages of a kind, under a path, a field, enough words, or what the digest calls significant; a search's new results; an agent's records — it starts an agent from a prompt you write (with the change filled in), re-crawls the changed pages, runs a monitor, or re-checks only the records of an agent monitor that were read on a changed page. Each has a wait between fires, a most a day and a monthly budget; **History** lists every fire and what it started, or why it was skipped, and **Preview** shows what the latest run would do.

Below them, what is queued and running for the workspace, each lane's queue depth and oldest wait, sparklines of pages per day.

In the dialog that keeps a search or an agent run, **Most credits a month** starts at about what a month costs, rounded up (3,412 becomes 3,500), so a month that runs past what was expected stops there; change it, or clear it for no cap. A search or agent run kept with **Save and schedule** opens at its own monitor page: its schedule, caps, who is told, and every run with what changed since the last one that answered. When an agent's answer is a list of records, the dialog offers **Follow each record**, named by a column you pick (the monitor page's **Records** line changes it later, and can write the records to a destination after each run). The page then shows the **Records** — every one, what moved, and the ones no longer there, each value with the page it was read on. A record one answer missed is shown as *withheld* rather than gone: it is gone only when two answers in a row lack it.

## Billing

The plan and what it includes, this month's meter (credits used of the allowance; the free tier's is a lifetime allowance), the **ledger** (every charge: a scrape, a map, a crawl's rows per engine with a note when renders were billed at the rung below, model passes, free lines for cached pages), the price list, and the billing portal (invoices, the card on file, plan changes). [Credits, plans and billing](/docs/billing) explains what a page costs and what each plan includes.

## Saved logins

Some sites show their content only to an account: a supplier's portal, a members' directory, a paid database, your own tools. A saved login lets a project, a scrape or an agent run read such a site signed in. It is switched on per deployment (MeshArc's own: ask us), and only owners and admins can save, see or delete one.

**Saving one.** Settings → Logins → **Add login**: name the site (`portal.supplier.com`), tick the terms, and press **Open the browser and sign in**. A real Chrome opens on MeshArc's servers and its screen is shown in the app: click, type, scroll, paste, use the address box or Back as in any browser, and sign in as you normally would — including a code from your phone. When you are in, press **Save login**. MeshArc keeps the signed-in session (the site's cookies and local storage), encrypted, and closes the browser; your password went into the site's own form and is never kept. The browser closes by itself after a few minutes, or sooner with nothing clicked or typed, and each person has one open at a time.

**Using one.** Pick it as **Saved login** under Configure → Fetch behaviour (a project) or in the playground's rail (a page). The agent needs nothing picked: it reads public pages signed out, as anyone sees them, and signs in where a page needs your account, with the saved login for that page's site. One message can read a site's public pages signed out and its account pages signed in, and use several logins; the page's URL decides which. **Saved logins** under Advanced in the agent's controls turns signing in off, or keeps it to one login, and each page read signed in says so in the message's steps ("signed in as …"). To have a page read signed in, say so in your message ("use my login for …"). A login is used on its own site and that site's subdomains, and never sent anywhere else. A page rendered with a login gets a browser of its own, never one another run shares.

**When it stops working.** Sites end sessions. When a run meets the site's login page again, the login is marked **expired**, the workspace's owners and admins get an email, and runs that use it stop — saying so — until it is saved again: **Sign in again** opens the live browser on its site, and what you save there renews that login, so the projects, monitors and runs that use it carry on. A run never quietly reads the site signed out instead. An agent answer, or an agent monitor's run, that needed a saved login it could not use says so under it — which login, and that its site was read signed out — whatever the answer itself says; on Automatic, a login saved again while the run goes on is used from then on. **Test** fetches the site's front page with the login and says whether the login page came back; **Expire** stops using one now; **Delete** removes it, and the projects that named it stop until another is chosen.

## Settings

Members and invites (with whether each has two-factor on, and the workspace's [two-factor rule](#two-factor-authentication)), API keys, [saved logins](#saved-logins), the rate ceiling, operator limits (what the plan and the operator allow: projects, pages per run, concurrent jobs, tier, rpm), the ladder (`/meta`), the proxy pool with per-exit parity and success, and usage by day and by engine with the blocked share.

---

# Configuration

Source: https://mesharc.dev/docs/configuration

# Configuration: every setting

A project's `config`, a one-shot `POST /crawl`, a `POST /scrape` (its `config` and `scrapeOptions`) and the playground all take the same settings with the same defaults. The Configure screen and the playground rail show them as the same 84 controls, so a request that names only what differs runs what the screen showed. `GET /meta` returns the full table.

Keys are given by their API name; the app's label follows in parentheses. **Default** is what you get when the key is absent.

## How a setting reaches the engine

| Where | How to set it |
|---|---|
| A project | `PATCH /projects/{id}` with `{"config": {...}}`, or the Configure screen |
| A one-shot crawl | `POST /crawl` with `config: {...}`, or the short names `limit`, `maxDepth`, `includePaths`, `excludePaths`, `useSitemap`, `sitemapMode`, `crawlMode`, `allowSubdomains`, `maxAge`, `maxTier`, `delay`, `concurrency`, `respectRobots` |
| A scrape | `POST /scrape` with `config: {...}`, or `scrapeOptions: {formats, actions, actionsOn, actionsPaths, onlyMainContent, includeTags, excludeTags, waitFor, waitMs, viewport, maxTier, jsonSchema, parseDocuments, minWords, languages}`, plus `maxCredits` and `formats` at the top level |
| The playground | The rail; the request sends what differs from the defaults |

On a one-shot `POST /crawl` or `POST /scrape`, unknown keys are ignored, not refused, so a newer SDK never fails on an option the API could do without. A project's `config` (`POST` and `PATCH /projects`, and the screens that send it) refuses a key that is not a setting with 422, naming it, so a misspelt key is never taken for one that was set. A value out of range is refused (400, or 422 from a project's `config`) with the range.

## Output formats

| Key | Meaning | Default |
|---|---|---|
| `formats` | Which bodies a page keeps: `markdown` (always), `text`, `rawHtml` (the page as served, for every page — 5–20× the storage), `cleanHtml` (after boilerplate, selectors and images, before markdown, without scripts and styles — selectors still resolve), `links` (every link with anchor text, resolved, typed internal / external / anchor), `raw` (status, headers, redirect chain and served body), `screenshot` (a JPEG of the viewport per page, ≤600 kB; forces a render, +1 credit), `json` (structured fields) | `["markdown", "links"]` |
| `json_schema` (JSON schema) | The fields the `json` format fills from what the page declares — head and Open Graph meta, JSON-LD, microdata, `<time>`, byline markup: `[{name, type, required}]`, types `string / number / date / boolean / url / array`. Names match loosely (`published_at`, `datePublished` and `date` are one fact). Unresolved is reported, never guessed | `[]` |
| `rows` (Read the page as records) | `{source, path, selector, columns, key, next, filters}`: a page that is a list — a board of roles, a table of recalls, a feed of posts — read as records instead of as one thing. `source` is `json` (a dotted `path` into the body, e.g. `data.children[].data`), `table` (the page's markdown tables, headings matched loosely) or `list` (a CSS `selector` matching one record). `columns` are typed as a JSON schema is; `key` is the column that names a record, or several when no single one does. `next: {path, param, max_pages}` follows a cursor the body hands out, for a feed nothing links to. `filters: {since_days, must_contain, drop}` keep records out and are counted. Rows are compared with the previous run's by key: added, changed (one event per column, with both values) and gone — gone withheld when the run reached under 90% of the site: such a row stays listed, marked `withheld` (one `withheld` event), until a run sees it again | `null` (off) |
| `llm_extract` (Fill missing fields with a model) | `{connection_id, instructions, only_missing, max_pages, max_chars, max_output_tokens}`: a Models connection fills the fields the page did not declare, from the markdown, cached by content hash; 1 credit per 1,000 tokens as the pipeline fee, the model bill is yours. `only_missing` true asks only for unresolved fields; `max_pages` caps calls per run (1–2,000), and the workspace's credits cap them too: the stage stops asking once what the crawl left is spent (the run's model tally says `stopped: credits`), while cached answers still fill their pages. `max_chars` (0 = as much of the page as the model's context holds) and `max_output_tokens` (0 = a budget that grows with the schema, an array field being a table) are ceilings for a workspace that would rather spend fewer credits on a long page: past `max_chars` the blocks naming your schema's fields are sent instead of the page's opening, and a reply that runs out of room is cut at its last complete value rather than lost | `null` (off) |
| `heading_style` | `atx` (`#` prefixes) or `setext` (underlines) | `atx` |
| `link_style` | `inline` (`[text](url)`), `reference` (footnotes), `strip` (text only — smallest) | `inline` |

## Content filters

| Key | Meaning | Default |
|---|---|---|
| `only_main_content` | Drop nav, header, footer and sidebars — the chrome that repeats on every page — and related-content blocks wherever they sit (related posts, "you may also like", recommendation widgets), so a menu edit or a new post elsewhere is not "the whole site changed". A sidebar inside the page's `main` or `article`, a sidebar or related block holding most of the page's words or the `main` or `article` itself, and anything `include_tags` names or `preserve_tags` keeps, stay | `true` |
| `strip_boilerplate` | Cookie banners, newsletter prompts, "was this helpful", share bars, back-to-top — by vendor markup and by shape | `true` |
| `strip_base64` | Inline data-URI images out of the markdown (their alt text stays) | `true` |
| `include_tags` | CSS selectors to keep exclusively; everything outside them is dropped | `[]` |
| `exclude_tags` | CSS selectors to drop; exclude wins over include | `[]` |
| `preserve_tags` | Selectors that survive every kind of pruning — a price table in a sidebar the chrome drop would take | `[]` |
| `min_words` (Minimum word count) | Off at 0. Set a floor and pages under it are recorded as **skipped** — stored, out of the change record. A page whose markup has a **shape** (a listing, a table, a form) is never under the floor, and the home page is never skipped. A short page is marked `warnings: ["short"]` on its row either way (0–5,000) | `0` |
| `languages` | Two-letter codes; a page outside the list is skipped. Detected from the `html` tag, headers or text; an undetectable one is never skipped | `[]` |

## Crawl scope

| Key | Meaning | Default |
|---|---|---|
| `max_pages` (Page budget) | The most pages one run fetches; the plan caps it (free 500 · Starter 5,000 · Growth 50,000 · Scale 200,000). A run that stops at its budget with URLs queued reaches only part of the site, and the diff withholds removals | `100` |
| `max_depth` | Link hops from the seed; 0 is the seed alone (0–10) | `3` |
| `breadth_per_level` | Cap on pages fetched at each depth; 0 is none. A cap silently leaves the rest of a level unread, which the link graph reports as reduced coverage | `0` |
| `respect_robots` | Obey Allow / Disallow (for `mesharc`, else `*`) and `Crawl-delay`; disallowed URLs are recorded as `robots`, never fetched | `true` |
| `allow_subdomains` | Follow links to other hosts under the same registrable domain | `false` |
| `include_paths` / `exclude_paths` | Globs on the URL path (`/docs/*`); excludes win; empty include means everything | `[]` |
| `sitemap_mode` (Sitemap source) | `auto` (robots.txt, the well-known paths, the seed page's link tags; indexes walked), `listed` (only `sitemaps`), `off` | `auto` |
| `sitemaps` | The sitemap URLs a `listed` project reads (up to 20, same site) | `[]` |
| `sitemap_include` / `sitemap_exclude` | Globs on a child file's URL or section label (`*press*`) that choose which files a crawl keeps | `[]` |
| `crawl_mode` (Crawl from) | `sitemap_first` (the selected sitemap URLs at depth 1, then links — the only mode that can tell an orphan from a page never reached), `sitemap_only` (exactly the selected URLs, nothing followed), `links` (seed and links; the sitemap still informs the link graph and the sitemap diff). A project that turns its sitemap off without naming a mode falls back to `links` | `sitemap_first` |
| `sitemap_incremental` | A scheduled run fetches only what the sitemap says moved (added, or `lastmod` newer than the last real fetch) and carries the rest over from the last full run at no fetch. A full run is forced every `sitemap_full_every` runs, when fewer than 90 % of entries carry `lastmod`, or when a carried page is over 30 days old | `false` |
| `sitemap_full_every` | Full run every N runs of an incremental project (1–365) | `10` |
| `crawl_delay_ms` | The pause between requests to one host; a robots.txt crawl-delay raises it, a rising refusal rate slows it, nothing lowers it (0–60,000) | `1000` |
| `concurrency` | Parallel fetches against the host on the http rungs once the first page has settled which rung works; the delay becomes the gap between rounds. A render stays one page at a time; a robots crawl-delay forces 1. 2–4 is safe on sites that do not rate-limit (1–8) | `1` |
| `url_list` / `feeds` / `patterns` | Sources beyond the seed, set on the [Sources screen](/docs/app#sources) (5,000 URLs; 20 feeds; 20 patterns, 5,000 expansions) | `[]` |

## Images

| Key | Meaning | Default |
|---|---|---|
| `image_mode` | `alt` writes the alt text in place of the image; `link` keeps `![alt](url)` with the URL made absolute | `alt` |
| `image_exclude` | Globs on the image path (`*/icons/*`, `*sprite*`) | `[]` |
| `image_min_width` / `image_min_height` | Drop images the markup declares smaller (tracking pixels, icons); only attributes are read, nothing is fetched | `0` / `0` |
| `image_max_kb` | Drop inline data-URI images above this | `0` |
| `alt_fallback` | When an image has no alt: `filename`, `caption` (the figcaption), `nearby` (surrounding text), `none` | `filename` |

## Fetch behaviour

| Key | Meaning | Default |
|---|---|---|
| `max_tier` (Max tier) | How far a refused page may climb; a page starts where the site is known to answer (the cheapest engine, on a site not read before) and climbs only when refused: `auto` (as high as the plan allows), `http` (1–2 credits; a walled page is recorded `TIER_LIMIT`), `browser` (a render when needed, 4), `stealth` (a residential exit 16, a solved widget 22, a rented fetch 40). The plan caps it (free and Starter: browser), and on any tier a climb never goes past the credits left | `auto` for a project made with New project, else `browser` |
| `render_js` (Render JavaScript) | `auto`: render only when the http rung comes back an app shell — and a render that only confirms a short page is billed as the http rung. `always`: every page, 4 credits. `never`: whatever the server sent (sets `max_tier` to http) | `auto` |
| `googlebot` (Ask as Googlebot) | Add Googlebot's identity to the engines this project may use. A claim about who is asking; sites that verify by reverse DNS are not persuaded; never reached for unless on | `false` |
| `user_agent` | Where the ladder starts: `default` (a rotating crawler identity), `chrome` (a real Chrome with its TLS fingerprint), `googlebot` (turns the Googlebot setting on) | `default` |
| `solve_captcha` | Have MeshArc's token solver answer a Turnstile / reCAPTCHA **widget on a form the steps submit**, before the steps run. Stealth tier; 22 credits when a token is bought. A widget on a page you only read needs no solving | `false` |
| `max_credits_per_page` (Max credits per page) | The most one page may cost; engines dearer than this are never tried, and a page only they could read is `TIER_LIMIT` at 0. 0: the tier is the ceiling. `maxCredits` on `/scrape` (0–100) | `0` |
| `max_credits_per_run` (Max credits per run) | The most one run may spend; a run that reaches it stops with `run cap`, keeping what it read. 0: no cap beyond what the workspace has left | `0` |
| `max_credits_per_month` (Max credits per month) | The most the scheduled runs may spend in a calendar month (UTC). A run due once it is spent is skipped (the project says so) and a webhook that asked for `budget.reached` hears it once that month; Run now is not held to it. 0: no cap | `0` |
| `wait_for_selector` | When rendering, wait up to 20 s for this node before reading; read what is there if it never appears | `""` |
| `wait_ms` | A flat delay after load on top of the settle; paid on every rendered page (0–60,000) | `0` |
| `actions` (Browser steps) | Steps after load and before reading: `click` (with `repeat: "until_gone"` and `max` for a Load-more button), `clickall`, `each` (over every element a selector matches, with nested `steps`), `type`, `select`, `press`, `wait` (ms or selector), `scroll` (bottom ×N or to a selector). Up to 40 leaf steps; a step that fails is noted on the page and the page is read as it stands; a page that takes steps is always rendered | `[]` |
| `actions_on` (Run steps on) | `start` (the seed plus the URL list and pattern URLs — a listing whose Load-more reveals the pages to crawl; a consent click the browser remembers), `paths` (pages matching `actions_paths`), `all` (every page — forces a render for the whole run, 4 credits a page) | `start` |
| `actions_paths` | Path globs for `paths` | `[]` |
| `fetch_timeout_ms` | Abort a fetch after this; recorded as a timeout (1,000–120,000) | `30000` |
| `fetch_retries` | Extra tries on a dropped connection or 500 / 502 / 503 / 504 / 400 / 408 / 409 / 430. A 429 is handled by the pacing; a 403 climbs the ladder (0–5) | `2` |
| `viewport` | `desktop` or `mobile` for renders | `desktop` |
| `use_proxy` (Proxy pool for http fetches) | Route the plain http fetches through MeshArc's datacenter proxy pool (each exit is checked against a direct fetch before it is trusted). No change to the price. The residential exit for renders is `max_tier: stealth`, a separate thing | `false` |
| `geo` | `auto` or a two-letter country: requests from that country through the residential exit; a requirement the ladder honours (stealth) | `auto` |
| `cache_max_age_hours` | Reuse a page any project of the workspace fetched this recently, at 0 credits, instead of fetching (`maxAge` on the verbs; 0–720) | `0` |
| `force_cache` | Whether a **scheduled** run may answer from the cache. Off, a scheduled run always looks | `false` |

## Advanced

| Key | Meaning | Default |
|---|---|---|
| Schedule (`schedule` on the project, not in `config`) | `manual`, `hourly`, `daily` (03:00 UTC), `weekly` (Mondays 03:00 UTC). Change detection needs two runs. A 100-page project is about 400 credits a month weekly, 3,000 daily | The app creates projects **weekly**; `POST /projects` and one-shot crawls: `manual` |
| Retention (`retention` on the project) | How long page content (HTML, text, markdown) and screenshots are kept: `30d`, `90d`, `1y`, `forever`; the project's two most recent runs, and any older run whose pages a newer run still reuses, keep theirs whatever it says. URLs, extracted fields, hashes and the change record are kept regardless; sitemap snapshots' URL lists follow the window too | `90d` (a one-shot crawl is deleted after a day unless kept; kept, `90d` unless `retention` is given) |
| `webhook_url` | Where the project's messages go (http(s)) | `""` |
| `webhook_events` | Which of `page.changed`, `run.finished`, `page.blocked`, `field.canonical`, `site.failing`, `site.recovered`, `page.orphaned`, `page.linked`, `sitemap.added / removed / updated / file_added / file_removed`, and the job events | `["page.changed", "run.finished", "field.canonical"]` |
| `webhook_include` | `["markdown"]`: `page.changed` carries each added or modified page's `markdown` beside its URL, up to about 1 MB of markdown a message; a page past that, or not stored, comes with `markdown: null`, `markdownOmitted` (`too_large` or `not_stored`) and a `contentUrl` to fetch it from. Subscriptions are sent the message without it. An edit here does not bump the config version | `[]` |
| `notify_min_words` | A modified page under this many changed words stays out of `page.changed`; added and removed pages always count | `5` |
| `chat_url` | A Slack, Discord, Microsoft Teams (Workflows), Mattermost or Google Chat incoming-webhook URL (https). The run record posts there when a run finds something — never when it compared clean — in the service's own shape, plus `site.failing` / `site.recovered`. Returned masked; a masked value sent back keeps the one stored, `""` removes it. An edit here does not bump the config version | `""` |
| `change_digest` | `{connection_id, focus}` — an `llm` connection of the workspace and a line on what the reader cares about. After each compared run with something in it, the model writes a paragraph on what changed and why it matters, up to five points and a `significant` flag, from the changed paragraphs, the added pages' openings, the removed paths and the field diffs. Carried by the run email, the chat message, `run.finished` and the change set. Charged like enrichment, through your own key | `null` |
| `login_id` | A [saved login](/docs/app#saved-logins) the runs read the site with, signed in: its id (`GET /logins`). It must be for this project's site (or a parent of it); it is used on that site and its subdomains only. A run whose login has expired or was deleted stops and says so — it never reads the site signed out instead. `""` reads signed out | `""` |
| `parse_documents` | Follow links to PDFs and Word (`.docx`) files and record each one's text as a document — a PDF as one markdown block per page, a Word file as its headings, paragraphs, lists and tables — with a word count and a hash, compared like a page (2 credits; a rendered one 4; up to 400 pages / 25 MB). Older Word (`.doc`), PowerPoint and Excel files are recognised and recorded as seen, not read (no reader ships yet). A scanned PDF (no text layer at all) is read by OCR, its first 30 pages, and its page says `extractMethod: "ocr"`; if OCR finds no text either, or fails, the page says so. Off, documents are never fetched — one met by content type is recorded as seen and not charged | `false` |

## Legacy aliases

`use_sitemap` (true unless `sitemap_mode` is `off`), `allow_browser` (false when `render_js` is `never`) and `keep_pages` are kept in step automatically for older clients.

## Things that follow from a setting

- `screenshot` in `formats`, or steps with `actions_on: all`, force `render_js: always`.
- `render_js: never` forces `max_tier: http`.
- `user_agent: googlebot` sets `googlebot: true`.
- `crawl_mode` other than `links` needs a sitemap: with `sitemap_mode: off` an explicit mode is refused, an implicit one becomes `links`.
- `sitemap_mode: listed` needs at least one URL in `sitemaps`; a sitemap or a source on another host is refused.
- A project's `max_pages` is capped by the plan's `max_pages_per_run`; `max_tier` by the plan's tier, on a scrape, the playground and a map as well. `auto` is the plan's tier.

---

# Connectors and destinations

Source: https://mesharc.dev/docs/connectors-destinations

# Connectors and destinations

**A connection** is a provider plus the credentials to reach it — a bucket and a key, a Postgres DSN, a Mongo URI. It belongs to the workspace, is named, and is **tested** before it is trusted (`POST /connections/{id}/test`). Secrets are kept apart from the rest of the config, encrypted at rest, and never returned by the API.

**A destination** binds a project — or, shared, every project or a chosen set — to a connection and says what to write: the target, which columns, the write mode, the key, whether every page goes or only the ones the change record says moved, and when. **A sync** is one destination writing one run, recorded with rows written, rows skipped and why, and the location.

Push is free: no credits are charged for writing.

## The connectors

**Verified** says whether MeshArc has run it end to end against a live instance; the ones marked *account* are written to the provider's current documentation and are confirmed with a real account on request.

| Provider | Kind | Fields | Secrets | What it writes | Verified |
|---|---|---|---|---|---|
| **Any LLM** (`llm`) | model | `base_url`, `model` | `api_key` (optional) | Not a destination: fills a project's JSON schema, writes a project's change digest, and can run the agent on your own key (`connectionId` on `POST /agent`, **Model** in the agent's options; the model must support tool calls, which **Test** checks). OpenAI, Anthropic, Gemini, Mistral, Groq, Ollama, vLLM and most others — the base URL of the chat-completions endpoint, the model name, your key | ✓ |
| **People Data Labs** (`pdl`) | data provider | — | `api_key` | Not a destination: the agent looks companies up with it (`dataConnections` on `POST /agent`, **Data providers** in the agent's options). Your key; People Data Labs bills each match, MeshArc charges no credits — see [Data providers](/docs/api#data-providers) | account |
| **Financial Modeling Prep** (`fmp`) | data provider | — | `api_key` | Not a destination: the agent reads a listed company's profile, quote, statements and key metrics with it. Your key and your plan's calls; MeshArc charges no credits | account |
| **Amazon S3** and S3-compatible (`s3`) | object store | `bucket`, `prefix`, `region`, `endpoint` | `access_key_id`, `secret_access_key` | One JSONL file per run under a dated prefix; set `endpoint` for MinIO, R2 or GCS with HMAC keys | ✓ (MinIO) |
| **Google Cloud Storage** (`gcs`) | object store | `bucket`, `prefix` | `access_key_id`, `secret_access_key` | Through the S3-interoperability endpoint with HMAC keys; one JSONL file per run | ✓ |
| **Azure Blob** (`azure`) | object store | `container`, `prefix` | `connection_string` | One JSONL blob per run under a dated prefix; the container is created if missing | ✓ (Azurite) |
| **PostgreSQL** (`postgres`) | database | `schema` | `dsn` | One row per page version; upsert with `ON CONFLICT` on the dedupe key; the table is created if missing | ✓ |
| **MySQL** (`mysql`) | database | `database` | `dsn` | One row per page version; `INSERT … ON DUPLICATE KEY UPDATE`; table created if missing | ✓ |
| **MongoDB** (`mongodb`) | database | `database` | `uri` | One document per page version, markdown inline; upsert on the key | ✓ |
| **ClickHouse** (`clickhouse`) | database | `url`, `database` | `user`, `password` | MergeTree ordered by `(url, crawled_at)`; upsert through ReplacingMergeTree; over HTTP | ✓ |
| **Google BigQuery** (`bigquery`) | warehouse | `project`, `dataset`, `location` | `service_account_json` | A load job per run; upsert stages then `MERGE`s. Needs `bigquery.tables.create/updateData` on the dataset | account |
| **Snowflake** (`snowflake`) | warehouse | `account`, `user`, `warehouse`, `database`, `schema`, `role` | password or key-pair (PEM) | A temporary table then `MERGE`; JSON columns as `VARIANT` | account |
| **Amazon Redshift** (`redshift`) | warehouse | `host`, `port`, `database`, `schema`, `user` | `password` | A temp table then `MERGE` on the key; JSON as `SUPER`; rows over the wire, not S3 `COPY` | account |
| **Databricks** (`databricks`) | warehouse | `server_hostname`, `http_path`, `catalog`, `schema` | `access_token` | A Delta table in Unity Catalog; upsert stages then `MERGE INTO` | account |
| **Pinecone** (`pinecone`) | vector index | `cloud`, `region`, `namespace`, `embedding_url`, `embedding_model`, `embedding_dims` | Pinecone key, embedder key | A serverless index with the embedder's dimensions; one vector per chunk, mapped columns as metadata; a page's chunks replaced by id prefix on upsert | account |
| **Qdrant** (`qdrant`) | vector index | `url`, `embedding_url`, `embedding_model`, `embedding_dims` | Qdrant key, embedder key | One point per chunk, mapped columns as payload, upsert per URL | ✓ |
| **Weaviate** (`weaviate`) | vector index | `url`, `grpc_port`, `embedding_url`, `embedding_model`, `embedding_dims` | Weaviate key, embedder key | A collection with self-provided vectors; one object per chunk; a page's chunks replaced on upsert. Cloud or self-hosted | ✓ |
| **pgvector** (`pgvector`) | vector index | `schema`, `embedding_url`, `embedding_model`, `embedding_dims` | `dsn`, embedder key | Chunk, embedding and the mapped columns in one table (`vector(dims)`); upsert per URL; the extension is created if missing | ✓ |
| **Kafka** (`kafka`) | stream | `bootstrap_servers`, `security_protocol`, `sasl_mechanism` | SASL username / password | One JSON message per row keyed on the URL, with `x-mesharc-run` and `x-mesharc-change` headers; the topic is created if missing; PLAINTEXT or SASL (PLAIN / SCRAM) | ✓ (Redpanda) |
| **HTTP webhook** (`webhook`) | stream | — | — | Not a connection: configured per project under Configure → Advanced — see [webhooks](/docs/api#webhooks) | ✓ |

The catalogue also lists a **Local disk** connector; it writes to MeshArc's own storage and exists for testing, not for your data.

**Vector indexes:** MeshArc chunks the markdown into paragraph-aligned pieces of about 350 words with a 40-word overlap; your OpenAI-compatible embeddings endpoint (`embedding_url` + `embedding_model`, dimensions declared) embeds them. Each chunk is a point with a stable id per `(url, chunk)`, the vector, and the mapped columns as payload with the chunk text and index — never the whole markdown.

## A destination

| Field | Meaning | Default |
|---|---|---|
| `connection_id` | Which connection | — |
| `target` | A table, a collection, a topic, or a path under the bucket / directory | required |
| `mode` | `upsert` (on the key) or `append`. A file or a message per run cannot upsert, so object stores and Kafka are recorded as `append` | `upsert` |
| `key` | What makes a row unique: `url`, `url+crawled_at`, `url+content_hash`. The key columns lead the row | `url` |
| `columns` | Which of MeshArc's columns to write, and in what order (every column is listed under The columns) | the default set |
| `incremental` | `true`: only pages the change record classed added, modified, removed, with field changes, or a baseline — so a destination receives what moved. `false`: every page of the run | `true` |
| `schedule` | `after_run` (fires when a run finishes) or `manual` (**Sync now**) | `after_run` |
| `scope` | `project` (one project's), `all` (every project of the workspace) or `projects` (a chosen set) — shared destinations carry `site`, `project_id` and `project_name` so one table holds many sites | `project` |
| `shape` | A filter, added columns and dropped columns — see Shaping | none |
| `enabled`, `name` | Pause without deleting; a label | `true`, `""` |

Up to 10 destinations per project and 20 shared ones per workspace; a sync writes at most 20,000 rows.

### The columns

| Group | Columns |
|---|---|
| identity | `url`, `site`, `final_url`, `crawled_at`, `run_id`, `project_id`, `project_name`, `content_hash`, `html_hash` |
| content | `markdown`, `text`, `raw_html`, `word_count`, `token_count` |
| fields | `title`, `meta_description`, `canonical_tag`, `robots_content`, `h1_tag`, `og_tag`, `twitter_tag`, `redirect_urls`, `response_status`, `fields` (the structured fields as JSON) |
| diagnostics | `block_reason`, `fetch_ms`, `depth`, `links`, `images`, `archive_path`, `language`, `found_via`, `first_referer`, `in_links`, `in_sitemap`, `orphan`, `unlisted` |
| change | `change_class`, `added_words`, `removed_words`, `changed_fields`, `reviewed_by` |

The default set: `url, site, final_url, crawled_at, run_id, project_id, content_hash, markdown, word_count, token_count, title, meta_description, canonical_tag, robots_content, h1_tag, redirect_urls, response_status, block_reason, depth, change_class, added_words, removed_words, changed_fields`.

## Shaping

A warehouse has dbt after the load; an application table or a vector index has nothing after the write, so what lands must already be the shape the reader wants. A shape is declarative — no code runs — and lives on the destination, so the same run can land raw in one place and shaped in another. `POST /destinations/preview` shows the shaped rows of the last run before you save.

```json
{
  "filter": [
    {"field": "status", "op": "eq", "value": "ok"},
    {"field": "path", "op": "glob", "value": "/products/*"},
    {"field": "fields.price", "op": "exists"},
    {"field": "word_count", "op": "gte", "value": 120}
  ],
  "map": [
    {"name": "price_cents", "from": "fields.price", "ops": ["number", "mul:100"], "cast": "int"},
    {"name": "category", "from": "url", "ops": ["path_segment:1"]},
    {"name": "source", "from": "=mesharc"}
  ],
  "drop": ["raw_html"]
}
```

- **filter** — conditions every row must pass, all of them (up to 20). Fields are the row's columns, plus `status`, `path` and `host` (visible to the filter even when not written), plus anything under `fields.` from the page's structured data. Operators: `eq`, `ne`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte`, `contains`, `not_contains`, `matches` (regex), `glob`, `exists`, `missing`, `empty`, `not_empty`.
- **map** — columns to add or replace (up to 60), each from a source field (or a literal with `=`) through a chain of up to 8 functions and a cast (`string`, `int`, `float`, `bool`, `date`, `json`). Functions: `trim`, `lower`, `upper`, `number`, `mul:N`, `date`, `host`, `path`, `path_segment:N`, `regex:pattern`, `split:sep`, `slice:a:b`, `replace:from:to`, `default:value`, `len`, `first`, `join:sep`, `json`, `not_empty`.
- **drop** — base columns to leave out after mapping.

What a shape cannot do belongs in the reader's own pipeline; the function set is small on purpose.

## One-off jobs

A destination does not need a project. `POST /scrape`, `POST /crawl` and the Playground (**Send to**) may each name one, and when the job is done its pages are written through that destination's columns, shape and sink — every page, with `change_class` of `baseline`, since a one-off has no run before it to compare with. A batch of scrapes is one write, however many hosts it spans. The write is on the destination's own sync history and on the job, under `destinationSync`.

A one-off crawl writes **only** where it names. A shared destination scoped to every project (`all`) does not receive one-off crawls: that scope is about the sites you watch, and a quick crawl turning up in that table would be a surprise. Kept as a project (`POST /crawl/{id}/keep`), it is written like any other.

## Syncs

`GET /destinations/{id}/syncs` (and the per-project form) lists every sync: when, which run (and which project's, for a shared destination), rows written, rows skipped and why (filtered out, unchanged, over the row cap), the location written, and an error when the write failed. A failed sync does not fail the run; the Destinations screen shows it and **Sync now** retries.

---

# The API

Source: https://mesharc.dev/docs/api

# The API

Base URL `https://api.mesharc.dev/api/v1`. The OpenAPI spec is at `https://api.mesharc.dev/docs`. The app uses these same routes; there is no private API behind it.

## Authentication

```
Authorization: Bearer mesharc_…
```

A key is issued under Settings → API keys (or `POST /me/keys`) and shown once. It carries scopes (`read` < `write` < `admin`), optionally a set of projects it may see, an expiry, and a rate limit. The app signs in with a session cookie instead and reaches the API through its own `/api/mesharc` proxy; the routes are the same.

A route that changes something needs `write` (the `member` role); keys, the rate ceiling, deleting a project, webhook secrets, connections and destinations need `admin`. A session whose email is not verified is refused those with `403 email_unverified`, and one in a workspace that requires [two-factor authentication](/docs/app#two-factor-authentication) without having it on is refused with `403 mfa_required`; a session with two-factor on that last showed its code more than ten minutes ago gets `403 mfa_stale` from the handful of actions that ask again (`POST /auth/2fa/verify` with a code clears it). API keys are unaffected by all three.

A key held to some projects sees only those: any other project is a `404`. The routes about the whole workspace — its [sub-accounts](#sub-accounts) (`/orgs/children`) and its [saved logins](#saved-logins) (`/logins`) — refuse it with `403 project_key`, and it may make only keys held to some of its projects (`POST /me/keys` without `projects` is `403 project_key`).

A key of a workspace with [sub-accounts](#sub-accounts) acts on one of them by naming it:

```
X-MeshArc-Tenant: <sub-account id>
```

The request is then the sub-account's, with the key's own scopes and rate limit. Only a key of that sub-account's own parent may do this, and only a key that is not held to some projects. A session is told to switch workspace instead (`400 tenant_refused`); a key held to some projects is refused with `403 tenant_refused`; an id that is not one of the key's workspace's sub-accounts — another workspace's, a sibling's, the parent's own, or any id sent with a sub-account's own key — is `404`, as anything else not yours is; and a suspended sub-account is `403 suspended`. Only the `404` is kept in the sign-in log, as a refused attempt; each use is counted per key, sub-account and day.

## Conventions

| | |
|---|---|
| **Job envelope** | Every asynchronous verb answers the same shape: `{id, kind, status, progress: {done, total, queued}, creditsUsed, counts, data: [...], next, cursor, stop, request_id, error}`. `status` is `queued` / `running` / `done` / `error` / `cancelled`; `next` is a ready URL for the page after this one. |
| **Pagination** | `?cursor=&limit=` — `limit` up to 1,000 (25 by default on a crawl's pages; up to 100 without bodies); `next` carries the cursor. A cursor that is no page of the list (past 2⁶³−1) is `400`. |
| **Idempotency** | `Idempotency-Key: <your key>` on `POST /scrape`, `/crawl`, `/map`, `/search`, `/agent`, `/agent/{id}/continue`, `/agent/{id}/follow-up`, `/monitors`, `/monitors/{id}/run` and `/orgs/children` returns the first answer for 24 hours, scoped to the route, so a retry after a dropped connection never starts a second job. A retry that arrives while the first request with its key is still being taken answers 409 with code `in_flight`: ask again shortly. `POST /projects/{id}/runs` needs no key: a second run while one is queued or running answers 409. |
| **Request ids** | Every response carries `X-Request-Id` (yours if you sent one) and the body's `request_id`; quote it to support. |
| **Credits** | `X-MeshArc-Credits` on a scrape's response says what it cost; every envelope carries `creditsUsed`. |
| **Rate limits** | `X-RateLimit-Limit`, `-Remaining`, `-Reset` per key; 429 when exceeded. |
| **Errors** | `{"error": "<what happened>", "code": "<what it means>", "request_id": "…"}`. Codes: `validation` (400 / 422), `unauthorized` (401), `plan_limit` (402 — a plan that does not include something, or credits spent), `forbidden` (403), `email_unverified` (403), `project_key` (403 — a key held to some projects, on a route about the whole workspace), `suspended` (403 — the workspace, or the sub-account named, is suspended), `not_found` (404 — including anything the key may not see; existence is never confirmed), `conflict` (409), `in_flight` (409 — the first request with this `Idempotency-Key` is still being taken), `sub_account` (409 — a billing action on a sub-account, whose billing is its parent's; or a sub-account making sub-accounts), `tenant_refused` (400 / 403 — `X-MeshArc-Tenant` from a session, or from a key held to some projects), `logins_off` (409 — saved logins are not switched on here), `login_expired` (409), `login_host` (400 — a login for another site), `too_large` (413), `rate_limited` (429), `unavailable` (503), `internal` (500). |
| **Time** | ISO 8601 UTC everywhere. |

## The verbs

### `POST /scrape` — URLs in, content out, no project

```json
{"url": "https://example.com/pricing", "formats": "markdown,links", "timeout": 60,
 "config": {"render_js": "auto"}, "scrapeOptions": {"onlyMainContent": true}, "maxCredits": 4}
```

One `url` waits for the page and returns it — up to `timeout` seconds (60 by default, 120 at most), then 202 with the id to poll, the fetch carrying on either way. A list of `urls` (up to 500) is always 202: grouped by host, fanned out, polled at `GET /scrape/{id}` (a row per URL with `httpStatus`, `verdict`, `credits`) or announced to `webhook_url` (`batch.finished`, signed with the secret returned once). `DELETE /scrape/{id}` cancels; `GET /scrape` lists recent ones. `formats` picks the bodies returned (`markdown,text,cleanHtml,rawHtml`). `maxCredits` caps what one page may cost. `destination` names one of the workspace's [destinations](/docs/connectors-destinations) — a database, a warehouse, a bucket — and the result is written there as well as returned: a single URL before it answers, so the response carries `destinationSync` (`status`, `written`, `skipped`, `location`, `error`); a batch once, when its last URL is in, reported on `GET /scrape/{id}`. The destination is checked before anything is queued: another workspace's is a 404, a paused one a 400.

A page row: `url, httpStatus, status, verdict, errorCode, shape, warnings, signals, method, tier, credits, billedAs, climbedTo, words, language, markdown, text, cleanHtml, html, links, fields, screenshot, head, reason, crawledAt`.

### `POST /crawl` — a one-shot crawl, no project first

```json
{"url": "https://docs.example.com/", "limit": 200, "maxDepth": 3,
 "includePaths": ["/docs/*"], "crawlMode": "sitemap_first", "maxTier": "browser",
 "scrapeOptions": {"formats": ["markdown", "links"]},
 "webhook": {"url": "https://you.example/hook", "events": ["crawl.completed"], "metadata": {"ref": "job-7"}}}
```

202 with the envelope; the crawl runs as a hidden, ephemeral project, kept for a day unless you keep it with `POST /crawl/{id}/keep`. `GET /crawl/{id}?cursor=&limit=&formats=` pages through its pages oldest first; `GET /crawl/{id}/page?url=` returns one of them in full (matched under either scheme and with or without a trailing slash); `DELETE /crawl/{id}` stops it; `POST /crawl/{id}/keep` `{name, schedule, retention}` promotes it to a standing project with its run already in place. The short names (`limit`, `maxDepth`, `includePaths`, `excludePaths`, `useSitemap`, `sitemapMode`, `crawlMode`, `allowSubdomains`, `maxAge`, `maxTier`, `delay`, `concurrency`, `respectRobots`) and `scrapeOptions` map onto the [configuration](/docs/configuration); `config` takes any key directly. The `webhook`'s secret comes back once as `webhookSecret`. `destination` writes the crawl's pages to one of the workspace's destinations when it is done — every page, as a baseline — and `GET /crawl/{id}` carries how the write went under `destinationSync`.

### `POST /map` — the site's URLs from its sitemap

```json
{"url": "https://www.gov.uk/", "search": "visa", "limit": 500, "sitemapMode": "auto", "timeout": 10}
```

Synchronous when the tree resolves within `timeout` (10 s by default, 60 at most), else 202 and `GET /map/{id}`. Rows: `{url, lastmod, changefreq, source, file, section}`; `totals` (`files, urls, indexes, blocked`), `robots`, `method`. `limit` up to 50,000 (5,000 by default); `search` filters on the URL. Costs 1 credit per sitemap file read — most sites are one file; gov.uk's 29 are 29.

### `POST /search` — web or news results for a query

```json
{"query": "rail funding consultation 2026", "limit": 5, "country": "GB", "lang": "en", "freshness": "week",
 "includeDomains": ["gov.uk"], "excludeDomains": [], "scrape": {"formats": ["markdown"], "maxCredits": 16}}
```

Google's results page, read through the ladder like any page, and Bing's when Google refuses. These are results pages, not a search API. You get organic results only: `{position, url, title, snippet, source, engine}`, with no ads, answer boxes or AI summaries. `sources` is `["web"]` (the default) or `["news"]`, one per search: news reads Google's News tab (Bing News when Google refuses), and each news result adds its `publisher` and `age` as the engine shows it (`"20h"`, `"1 day ago"`). `limit` is 1–10 (one results page), and `page` (1–10) reads further in: page 2 is results 11–20, numbered from 11, each page a search and a charge of its own. `freshness` is `hour`, `day`, `week`, `month` or `year`; domains become `site:` operators. Waits like `/scrape` (`timeout`, 60 s by default, 120 at most), then 202 and `GET /search/{id}`. With `scrape`, the results' URLs are scraped as a batch and each result gains its `page` as it lands; the search reads `running` until they are in. The results page costs what its rung costs, typically a render (4 credits) for Google and 1 for Bing, and shows on the billing page as a search. A refused results page is free: `blocked` means every engine refused, and `attempts` says why. The same search within the hour (ten minutes for news) is answered from the last one at no charge (`cached: true`). Reading a results page is the one documented exception to robots.txt; the pages a search scrapes obey it as usual. Add `"webhook": "https://you.example/hook"` (or `{url, events, metadata}`) and the search's end is posted to it once, signed with the `webhookSecret` the `POST` returns once: `search.completed` when its results are in (before the pages a `scrape` reads, which `GET /search/{id}` shows as they land), `search.failed` when every engine refused it or it failed, an answer from the cache included. The data is `{search: {id, query, status, results, cached, error}}`, `results` being how many; `GET /search/{id}` has them. With `"timeout": 0` the `POST` answers at once with the id, so a caller that cannot wait half a minute hears from the webhook instead. Takes an `Idempotency-Key`.

### `GET /search` — past searches

`?limit=25&cursor=&q=` → `{data, next}`, newest first. Each row is `{id, query, params, status, engine, rung, cached, resultCount, domains, scraped, creditsUsed, error, createdAt, finishedAt}`: `domains` is the first three result hosts, and `creditsUsed` includes the scraped pages. `q` keeps searches whose query contains it, in any case. Reading the list fetches nothing and costs nothing; `GET /search/{id}` opens one as it was stored. Searches are kept 7 days, but a scraped page's text only 1 day. After that the result's `page` reads `{"status": "expired"}`.

### `POST /agent` — ask for data, get it back with its sources

```json
{"prompt": "List each competitor's pricing plans with the monthly price and what each plan includes",
 "urls": ["https://example.com", "https://example.org"],
 "schema": {"type": "object", "properties": {"tiers": {"type": "array", "items": {"type": "object",
   "properties": {"company": {"type": "string"}, "plan": {"type": "string"}, "price_month": {"type": "number"}}}}}},
 "maxCredits": 2000, "maxSteps": 40, "allowedDomains": ["example.com", "example.org"]}
```

An agent searches the web, reads the site maps and pages it needs through the same ladder as a crawl, and answers. When what it needs is behind a "Load more" button, a tab, a filter or an endless scroll, it renders the page with browser steps (click, scroll, wait, select; never typing, so it fills no form and logs in nowhere), at most three renders of a page per run, each a step in the trace with its actions and credits. With `schema` (a JSON Schema of type object or array), `data` is JSON in that shape; without one, `{text}`. `sources` lists the pages it read, and with a schema `fieldSources` says where each value came from: its path in `data` (`plans[0].price`, or `[2].name` when the answer is a list) to the `url` and `pageId` of the page it was read on. Only pages the run read are named; a value without an entry had no page to point to. `loginNotices` lists the saved logins the run needed and could not use — `{host, loginId, name, status}`, `status` `expired` or `deleted` — so those sites were read signed out, whatever the answer says; on `auto`, a login saved again while the run goes on is used from then on. It runs in the background: 202 with `next`, then poll `GET /agent/{id}` until `status` is not `queued` or `running` (or pass `timeout`, up to 120 s, to wait). Statuses: `done`, `credit_limit` (stopped at `maxCredits`; `data.partial` holds what it had), `error`, `cancelled`, and `expired` once its results are past their keep date (then `GET` answers 410).

Pages are charged as they are read, at the rung that answered (refused pages are free); model tokens at the model provider's price plus 20%, in credits (the billing page shows the rate per million tokens). `maxCredits` (2,000 by default) caps the run; a budget too small for even one model call (one costs tens of credits) stops at once, at `credit_limit`, with nothing spent. `allowedDomains` keeps it on those sites (and their subdomains; `*.example.com` reads as `example.com`). Each entry must be a domain, or the request is refused (400) — never quietly dropped, which would leave the run with no limit; the same holds for a search's `includeDomains` and `excludeDomains`. **Without `allowedDomains` the run has no domain limit**: it may read any site it finds, and a page it reads could try to send it elsewhere. That is a deliberate default for the API, so a run can follow the web; pass your starting URLs' domains to hold it to them. (The Playground does that by default and warns when a run has no limit.) The agent treats page text as data, never as instructions, every page it reads is in its trace, and `maxCredits` caps what any detour can cost. Takes an `Idempotency-Key`.

- `GET /agent/{id}/trace?after=N` — the run's steps after step N (model calls, searches, pages read with their rung and credits, maps, waits), `{data, last}`; poll with `after=last`.
- `GET /agent` — runs, newest first, `{data, next}`; `status` keeps one status, `model` one model's runs (as a run names it, e.g. `openai:gpt-5.4-mini`), and `since` / `until` (an ISO 8601 date or date-time) the runs made from or before then.
- `DELETE /agent/{id}` — cancel: a queued run stops at once, a running one before its next step. What it used stays charged.
- `POST /agent/{id}/continue` — carry on a run that stopped at its credit limit, with a new `maxCredits` (and `maxSteps`; the stopped run's when left out). It starts a new run on the same thread (`continuesRunId`) that resumes where the stopped one was, keeps the pages it read (not paid for again), its saved login and its data providers (each checked again), and answers in full; the stopped run then names it in `continuedBy`. A run can be continued once; 409 for a run that did not stop at its limit, 410 past its keep date. A run's webhook carries over.
- `POST /agent/{id}/follow-up` — ask a follow-up in a conversation. The body is a `POST /agent` request (its own `prompt`, `urls`, `schema`, `maxCredits`, `maxSteps`, `allowedDomains`, `connectionId`, `logins`, `dataConnections`, `webhook`, `timeout`); the new run is on the same thread (`threadId`), names the message it follows in `followsRunId`, and that message names it in `followedBy`. It answers after everything asked and found in the conversation before it, and may read again, free, any page the conversation read — such a read is a `fetch` step with `reused: true` and 0 credits, and the page is one of the follow-up's sources. `{id}` is the conversation's latest message, finished: 409 while it works, for an earlier message (the error names the latest), or past 30 messages; 410 past its keep date. Each follow-up pays for its own work, and restarts the conversation's 7 days.
- While a run is `running`, `GET /agent/{id}` (and each run of a conversation) carries `doing`: the step under way now, `{kind, text, url, since}` — `kind` is `model`, `search`, `fetch`, `render`, `map`, `extract` or `lookup`. The trace has a step once it has ended; `doing` is what it is doing before then. `null` otherwise.
- `GET /agent/conversations` — conversations, the latest message first: `{id (the thread, its first run's id), title (its first question), messages, status and lastRunId (its latest run), model, creditsUsed, createdAt, updatedAt, expiresAt}`; `q` keeps those whose first question holds it, `limit` and `cursor` page. A table's rows are not listed. `GET /agent/conversations/{id}` adds `runs`: every run in it, oldest first, each as `GET /agent/{id}` answers.

An agent run can read members-only pages with the workspace's [saved logins](#saved-logins), none of which needs picking: `"logins"` says which it may use — `"auto"` by default, every active saved login, or `"none"`, or a list of login ids (at most 20; `"loginId"`, one id, is the older form of a list of one). The agent reads pages signed out, as the public sees them, and signs in page by page where a page needs the account: what only the account shows, or a page that asked it to log in. A public page stays signed out, because a site may show a signed-in visitor something else or send it away from a public page; one run can read a site's public pages signed out and its account pages signed in. The page's URL picks the login: a page on a login's site, or one of its subdomains, is read with it — the most specific when two fit, a login for `console.example.com` before one for `example.com`. A login is never sent to another site. Each page read signed in names its login in the trace (`login`, `loginHost`), and a page the site answered with another names that one (`finalUrl`). A page that shows the site's login page signed in is reported as needing a login and that login is marked expired; one met signed out says when the run may read it signed in, or when a saved login for the site exists but has expired, or was not one the run named. An agent monitor's run reads the pages its last run read signed in so again, unless the agent decides otherwise, so a value is not compared with what the public sees. A list naming a login that is expired, deleted or not this workspace's is a 400 when the run is asked for; `"auto"` uses no login while logins are switched off.

To let it look up company or financial data, pass `"dataConnections"`: up to two of the workspace's [data providers](#data-providers), one per provider. Each gives the run its lookup tool, on your own key; MeshArc charges no credits for a lookup.

To run on your own model, pass `"connectionId"`: one of your workspace's LLM connections (any OpenAI-compatible chat endpoint, its model and your key; see Connectors). Your provider bills the tokens to your key, and MeshArc charges 1 credit per 1,000 tokens, as for every model stage on your key, plus the pages read. The model must support tool calls.

Add `"webhook": "https://you.example/hook"` (or `{url, events, metadata}`) and the run's life is posted to it, signed like every MeshArc webhook with the `webhookSecret` the `POST` returns once: `agent.started`, `agent.action` (the steps since the last one, at most every 5 s, with `progress`), then one of `agent.completed` (with `data`, `sources` and `creditsUsed`, as `GET /agent/{id}` answers; a run stopped at its limit is completed with status `credit_limit`), `agent.failed` or `agent.cancelled`. All five by default.

A run, its answer, its steps and its webhook messages are kept for 7 days.

Pass `"proposals": true` and the agent may propose things to watch, rather than only say so: crawling a site on a schedule, a search kept and run on a schedule, or a question asked again on a schedule — each with why and about what it would cost a month. Nothing is made, scheduled or charged: a proposal waits for a member to approve it (see `GET /proposals`). Without it, the agent has no way to propose.

A `pattern` in a `schema` is at most 200 characters, and a pattern that nests repetitions (`(a+)+`, `(\w+\s?)*`) is refused with a 400: answers are checked against it, and that shape can take exponential time on some text. Ordinary patterns — an email, dotted names such as `[a-z]+(\.[a-z]+)*`, a phone number — pass.

### `POST /agent/batch` — the agent over every row of a table

```json
{"csv": "Company,Website\nAcme,acme.com\nGlobex,globex.com\n",
 "prompt": "What does {{Company}} charge for its cheapest plan?", "urls": ["{{Website}}"],
 "schema": {"type": "object", "properties": {"plan": {"type": "string"}, "price_month": {"type": "number"}}},
 "maxCreditsPerRow": 300, "concurrency": 3, "name": "Competitor prices", "webhook": "https://you.example/hook"}
```

One agent run a row: the prompt and starting URLs filled from the row's columns, written as `{{column}}`, and the answer — shaped by `schema`, a JSON Schema of type object — the row's new columns. Send the table as `csv` (its first line names the columns; a cell like `acme.com` in a URL is read as `https://acme.com`) or as `rows`, a list of objects. Your plan sets how many rows a batch may have: 25 on the free plan, 200 on Starter, 500 on Growth, 1,000 on Scale.

Everything is checked before anything runs: every `{{column}}` must name a column, the prompt must name at least one (or the row must start at its own URL), and the schema is checked once. Then each row is checked as `POST /agent` would check it; a row that fails — its prompt came out empty, a cell is not a URL — is kept as `invalid`, with the reason, and never run or charged. A table where no row can run is a 400.

- **Where a row may go:** without `allowedDomains`, each row is held to its own URLs' sites, subdomains included (a row starting at `www.acme.com` may read `shop.acme.com`); a list holds every row to it; `[]` lifts the limit. A row with no URL goes where its search leads.
- **A few at a time:** `concurrency` rows run at once (3 by default, 10 at most, and never more than your plan's agent slots). As each row ends, the next starts.
- **What it costs:** `maxCreditsPerRow` is required: each row's run stops there and pays only for what it used. The batch never spends more than rows × that cap, or your balance when the batch started if that was less; a row it can no longer afford is `skipped`, free, saying why.
- **Data and logins:** `dataConnections` and `logins` (or `loginId`) are as `POST /agent` takes them, checked once for the batch and given to every row's run.
- **Before you start:** `POST /agent/batch/preview` with the same body checks everything and answers what would run — `valid`, `invalid` and the first invalid rows with their reasons, the first prompts as their rows will ask them (with their starting URLs and sites), and an `estimate`: about what a row costs (your last batch rows' average, or the cap before you have any), the total, the most it could be, and your balance. It starts nothing and spends nothing.

It answers 202 with the batch: `counts` by row status, `budget`, `creditsUsed`, `averagePerRow`, `next`. Takes an `Idempotency-Key`; with a `webhook`, `webhookSecret` comes back once and `agent.batch.finished` is posted once, when the last row has ended, with the totals.

- `GET /agent/batch/{id}?offset=&limit=&status=` — the batch and a page of its rows in table order (`limit` up to 200): each row's `values`, `status` (`pending`, `queued`, `running`, `done`, `credit_limit`, `error`, `cancelled`, `invalid`, `skipped`), `data`, `fieldSources`, `sources`, `creditsUsed`, `budget`, `error` and `runId` (the row's own run, at `GET /agent/{runId}`). `more` names the next page; `status` keeps one status.
- `GET /agent/batch/{id}/export?format=csv|jsonl` — each row's number (`row`, from 1) and `status`, then the table as you sent it, each answer value a new column (one named like an input column becomes `<name> (answer)`), then `creditsUsed`, `sources`, `error` and `runId`. In CSV, a cell that would start a spreadsheet formula is written as text.
- `GET /agent/batch` — batches, newest first. A batch's rows are not listed under `GET /agent`.
- `DELETE /agent/batch/{id}` — cancel: rows not started are cancelled, running ones stop before their next step. What rows used stays charged; their answers are kept.

A batch and its rows are kept as long as its runs: 7 days, then 410.

### `POST /monitors` — keep a search or an agent request and run it on a schedule

```json
{"kind": "search", "schedule": "daily", "request": {"query": "acme pricing", "limit": 10},
 "name": "Acme on Google", "webhook": "https://you.example/hook", "baselineId": "<a finished search's id>",
 "budget": {"perMonth": 500}, "alerts": {"email": true, "chatUrl": "https://hooks.slack.com/services/…"},
 "addToProject": "<a project id>"}
```

A monitor runs the same `POST /search` or `POST /agent` body (`request`, checked as those endpoints check it) on its `schedule`: `hourly`, `daily` (03:00 UTC) or `weekly` (Mondays, 03:00 UTC). Each run is an ordinary search or agent run, charged as one and listed with your others. When it ends, it is compared with the last run that answered:

- **A search:** results are matched by URL (no fragment, no trailing slash, `www.` the same site), giving `new` and `dropped` results (`{position, url, title}`, `position` may be null) and `moved` ones (`{url, title, from, to}`). When another engine answered than last time (Google refused and Bing read it), engines rank differently by nature: new results are still reported, but drops and moves are not (`comparable: false`). A monitored search reads its results page only, so `scrape` is refused.
- **An agent run:** the answer is compared value by value. Each change is `{path, kind, before, after, source}`: `kind` is `added`, `changed` or `removed`, and `source` is the page the value was read on. A list of records is matched by the field that names each record (`id`, `name`, `title`, `url`, `plan`, `sku`, `product`), so the same records in another order are no change. Text is compared with its spacing and case ignored.

The first run that answers is the baseline. `baselineId` names a finished search or agent run with the same request to use as that first run. For an agent run, budget and steps may differ. For a search, `scrape` may differ, but its results `limit` must be at least the monitor's (a search that read fewer results is refused: the ones it stopped short of would read as new). In the app, "Save and schedule" on a search and "Keep and re-run" on an agent run send the one on screen, so keeping it costs nothing. A run that continued another (`POST /agent/{id}/continue`) is not offered there: its `maxCredits` is only the top-up it was given, so run from scratch on a schedule it would stop at its credit limit each time. Nor is a follow-up (`POST /agent/{id}/follow-up`): it asks about the conversation before it, which a monitor's run would not have. Without it, the first run starts when the monitor is made. A run that did not answer (a blocked results page, an agent stopped at its credit limit, a failure) is withheld: kept and shown, but nothing is compared with it and nothing is announced. A slot runs once. A workspace with nothing left to spend skips the slot, and the run says so (`status: "skipped"`).

`alerts` tells people too: `email` mails the workspace's members who take run emails, and `chatUrl` (a Slack, Discord, Microsoft Teams, Google Chat or Mattermost incoming webhook, https) gets the change in that service's own shape -- the new, dropped and moved results, or the values added, changed and removed. The chat URL comes back masked; a masked value sent back keeps it. `addToProject` (a search monitor's) names a project of the workspace: the results of the first run, and each later run's new results, that are pages of that project's site are added to its listed URLs, so the next crawl reads them (`addedToProject` on the run says how many). A monitor's searches are kept 90 days, not a week.

`sentiment` (a news search monitor's: `request.sources` is `["news"]`) scores each run's new results with one of the workspace's LLM connections (see Connectors): `{"connectionId": "<an llm connection>", "only": ["negative"]}`. Each new result gains `sentiment` — `positive`, `neutral` or `negative` — and `sentimentReason`, a line on why, read from its title, publisher and snippet (the first 20 new results of a run; a baseline has none). The run's `diff.sentiment` says `{model, scored, of, error}`. The tokens go to your provider on your key, and MeshArc charges 1 credit per 1,000, counted in the run's credits (so a monthly cap sees them). `only` keeps the chat channel and email to runs with a new result of those tones; the webhook hears every change, with the tones. A model that fails leaves its `error` on the run and the change is announced unscored, to everyone. `{}` turns it off.

`budget` caps what it spends, in credits (0 or left out is no cap). `perMonth`: once its runs this calendar month (UTC) have cost that much, a due run is skipped (`status: "skipped"`, `error` saying the cap is reached) until the next month or a higher cap; Run now is not held to it. `perRun` is an agent monitor's: each run's `maxCredits` is the smaller of the request's and this (a monitored search reads one results page, 1 to 4 credits, so `perRun` is refused for one).

With a `webhook`, a run that changed something posts `search.changed` or `agent.changed` once (and a slot skipped for the monthly cap posts `budget.reached`, once that month), signed with the `webhookSecret` returned once (here, or by the `PATCH` that sets a new one). The data is `{monitor: {id, name, kind}, run}`, with the run as `GET /monitors/{id}/runs` shows it. Takes an `Idempotency-Key`. 201 with the monitor and its first run.

**An agent's answer as rows.** When an agent monitor's answer is a list of records — each competitor's plans, the open roles, the stockists — `rows` follows each record on its own, as a project's rows follow the records its pages hold:

```json
{"kind": "agent", "schedule": "daily",
 "request": {"prompt": "List Acme's plans with their monthly price", "urls": ["https://acme.example/pricing"],
             "schema": {"type": "object", "properties": {"plans": {"type": "array", "items": {"type": "object",
               "properties": {"name": {"type": "string"}, "price": {"type": "number"}}}}}}},
 "rows": {"path": "plans", "key": ["name"], "confirmAfter": 2}, "destination": "<a destination id>"}
```

`path` is the answer's list (left out: the answer itself when the schema is an array, or its one list of records); `key` the column or columns that name a record (left out: the first of `id`, `name`, `title`, `url`, `plan`, `sku`, `product`); the columns are the list's own properties, typed from the schema; `rows: true` takes every default. Each answered run then says which records are `added`, which `changed` (one entry per column, with both values) and which are gone — a run's `diff` lists them, `summary` counts them (`1 added · 1 changed · 1 withheld`), and the webhook also posts `row.changed` with the records that moved.

An agent cannot say how much of the web it saw, so a record missing from one answer is not gone: it is **withheld** — still listed, marked `withheld`, and recorded once as a `withheld` event — and it is gone only when `confirmAfter` answers in a row (2 by default, 1 to 10) lack it. An answer that lost half the records or more counts as no miss at all, and an answer that did not finish (its credit limit, a failure) changes nothing. Each value keeps the page it was read on (`sources`); the same value read from another page is not a change. A new `path` or `key`, or `rows: {}`, starts the records again (the next answer is their baseline).

- `GET /monitors/{id}/rows?active=&q=&limit=&offset=` — its records as they stand: `{key, attrs, url, firstSeen, lastSeen, active, withheld, misses, sources}`; `active=false` lists the records gone. `q` searches the records' values (not their column names), in any case; `total` counts its matches, and `truncated: true` says the search stopped at its scan limit (100,000 records), so `total` is of those it read.
- `GET /monitors/{id}/rows/events?run_id=&kind=` — what moved, newest first; `kind` is `added`, `changed`, `gone` or `withheld`, `run_id` a monitor run's id.
- `GET /monitors/{id}/rows/{key}` — one record, its sources, and every event that touched it.
- `GET /monitors/{id}/rows/export?dataset=rows|row-events&format=csv|jsonl` — as a file.

With a `destination` (any of the workspace's), each run's records — the ones it saw or called gone — are written there after the run: `monitor_id`, `monitor_name`, `key`, `label`, `state` (`active` / `gone`), `change` (`added`, `changed`, `gone`, `same`, or `baseline` on the first), `first_seen`, `last_seen`, `run_id`, and `record` and `sources` as JSON text. An upsert destination keeps one row per monitor and record; an append one gets each run's. The write is on the destination's history and on the run (`destinationSync`).

- `GET /monitors?kind=` — the workspace's monitors, newest first, each with `lastRun`, `lastRunAt`, `nextRunAt`, its `budget` (`{perRun, perMonth}`, 0 is none) and `spentThisMonth`. At most 200 per workspace.
- `GET /monitors/overview?kind=crawl|search|agent` — everything on a schedule in one list: crawl projects with a schedule (a manual project is not listed) and every monitor, `{data, counts}`. Each row is `{kind, id, name, detail, schedule, status, health, nextRunAt, lastRunAt, lastRun: {at, status, summary, changed, baseline, withheld, error}, skipped, spentThisMonth, estimate: {perMonth, upTo, capped, basis}, caps: {perRun, perMonth}, alerts: {webhook, email, chat}}` — `detail` is the project's host or what the monitor asks, `estimate` about what a month costs held to the monthly cap, `alerts.chat` the chat service or `""`. Active first, soonest next run first; paused last. A key held to some projects sees those, and no monitors.
- `GET /monitors/{id}` · `PATCH /monitors/{id}` `{name?, schedule?, status?, webhook?, budget?, alerts?, addToProject?, rows?, destination?, sentiment?}` — `status` is `active` or `paused`; `webhook: ""` removes it; `budget` and `alerts` are replaced whole (`{}` removes them); `addToProject: ""` stops feeding the project; `rows: {}` stops following records (and forgets them); `destination: ""` stops writing them.
- `POST /monitors/{id}/pause` · `POST /monitors/{id}/resume` — a paused monitor is not scheduled; a run under way finishes. Resumed, a slot that passed meanwhile runs once.
- `POST /monitors/{id}/run` — run it now, paused or not; the schedule counts on from this run. 409 while a run is under way.
- `GET /monitors/{id}/runs?limit=&cursor=` — its runs, newest first: `{id, monitorId, kind, trigger, status, refId, credits, answered, changed, summary, diff, error, loginNotices, createdAt, finishedAt}` (`loginNotices`: an agent monitor's run, the saved logins it could not use). `refId` is the run's own search or agent run (`GET /search/{refId}`, `/agent/{refId}`, kept 7 days). The comparison is kept with the run, for the last 200 runs.
- `DELETE /monitors/{id}` — the monitor and its runs; answers `{id, deleted: true}`. The searches and agent runs it made stay in your history; an agent run under way is stopped.

### `POST /triggers` — a change or a new result starts work

```json
{"name": "Pricing watch", "source": {"kind": "project", "id": "<a project id>"},
 "when": {"kinds": ["modified"], "paths": ["/pricing*"], "minWords": 10},
 "action": {"type": "agent", "prompt": "This page changed: {{url}}
{{changes}}
What changed, and does it matter?", "maxCredits": 300},
 "limits": {"cooldownMinutes": 60, "maxPerDay": 10, "perMonth": 2000}}
```

A trigger watches a project's change records or a monitor's runs, and when a finished run matches `when` it does one thing (`action.type`):

- `agent` — starts an agent run from `prompt`, a template: `{{url}}`, `{{before}}` and `{{after}}` are the first change's, `{{changes}}` lists every change (a line each), `{{source}}` is the project's or monitor's name. The page text (`{{before}}`, `{{after}}`, `{{changes}}`) is not spliced into the instruction: the prompt names it, and it follows in `<page-data>` blocks marked as data, which the agent is told never to take as instructions. It starts on the changed pages and is held to the project's site (a monitor's: the changed pages' sites) unless `allowedDomains` says otherwise; when no site can be told, the fire is skipped rather than run with no limit (`allowedDomains: []` asks for no limit on purpose). `maxCredits` (300 by default), `maxSteps`, `schema` and `connectionId` are as `POST /agent` takes them, checked when the trigger is made.
- `recrawl` — re-crawls the pages that were added or changed (a project's).
- `monitor` — runs a monitor now (`monitorId`).
- `recheck` — for a project's change: re-checks only the records of an agent monitor followed as rows (`monitorId`) whose value was read on a changed page. Its prompt is narrowed to those records and its answer applied to them alone — the records it did not look at are not missing. A change no record was read on starts nothing.

`when`, for a project: `kinds` (`added`, `modified`, `removed`; all by default), `paths` (globs: `/pricing*`), `fields` (head fields such as `title` or `canonical` — a field change rather than a page change), `minWords` (words added and removed on a changed page) and `significant` (only when the change digest called the change significant). For a monitor: `events` — a search's `new_result`, `dropped_result`, `moved_result`; an agent's values or records `added`, `changed`, `removed`. Baselines and re-baselines never fire.

Each finished run fires a trigger at most once, read about a minute after it ends, oldest first. A trigger looks from when it was made (or resumed): what came before is not news. A run that a trigger started never fires a trigger itself, so triggers do not chain. `limits`: `cooldownMinutes` between fires (60), `maxPerDay` (10, in any 24 hours), `perMonth` credits (what the runs it started cost this calendar month; 0 is no cap — an agent action's `maxCredits` is held to what is left). A match a limit holds back is recorded as a skipped fire, saying why. Runs a trigger started say so: `triggeredBy` on a monitor run, `TriggeredBy` on the agent or crawl run.

- `GET /triggers?source_id=&target_id=` — triggers, newest first, each with its `when`, `action`, `limits`, `spentThisMonth` and `lastFire`.
- `GET /triggers/{id}` · `PATCH /triggers/{id}` `{name?, when?, action?, limits?, status?}` — `status` is `active` or `paused`; a limit left out keeps its value.
- `GET /triggers/{id}/fires` — its firing history, newest first: `{at, status (fired, skipped, error), reason, sourceRunId, matched, items, refKind, refId, credits}`; `refKind` is `agent`, `crawl` or `monitor_run`. Kept 90 days.
- `POST /triggers/{id}/preview` — what the source's latest run would match, and the prompt an agent action would send; fires nothing.
- `DELETE /triggers/{id}` — the trigger and its history. Deleting a project or a monitor deletes the triggers that watch it or run it.

### `GET /proposals` — what agents proposed to watch

An agent run started with `"proposals": true` may propose a crawl project, a search monitor or an agent monitor. Each proposal is checked when it is made — a private address, a site the workspace already crawls, a search or question a monitor already keeps, the same thing already waiting, more than 5 from one run — and the agent is told why one was refused.

```json
{"data": [{"id": "…", "kind": "project", "name": "acme.test", "what": "https://acme.test/", "schedule": "daily",
  "reason": "You asked to keep an eye on Acme's site.", "estimate": {"perRun": 100, "perMonth": 3000, "upTo": true, "basis": "…"},
  "spec": {"seed": "https://acme.test/", "name": "acme.test", "schedule": "daily", "config": {"max_pages": 100}},
  "runId": "…", "runPrompt": "…", "status": "proposed", "made": null, "createdAt": "…", "decidedAt": null}]}
```

`status=proposed` (the default) lists those waiting for a decision, `status=approved` the approved, `status=` all; `run_id` keeps one run's. `spec` is what approving makes, as `POST /projects` or `POST /monitors` would take it. Proposals are kept 30 days. A key held to some projects sees none.

- `POST /proposals/{id}/approve` — member and up. Makes it: the project (as `POST /projects` makes one: its pages are discovered now, and it crawls at its first scheduled time — the next 03:00 UTC for daily — or when run; 402 at the workspace's project limit), or the monitor (its first run starts now; 402 with no credits left). Answers `{proposal, made: {kind: "project" | "monitor", id}}`. Once: 409 when it was decided already.
- `POST /proposals/{id}/dismiss` — member and up. Deletes one still waiting; 409 for one approved.

### `GET /estimate` — about what scheduled work costs a month

`?kind=crawl|search|agent&schedule=hourly|daily|weekly|manual`, and what to price: `projectId` or `monitorId` for one that exists, or for one not made yet `maxPages` and `maxTier` (a crawl) or `maxCredits` (an agent). Answers `{kind, schedule, perRun, runsPerMonth, perMonth, capped, cap, upTo, basis}`: runs a month (hourly 730, daily 30, weekly 4.35) times what a run costs, held to `monthCap` (or the project's or monitor's own monthly cap). Something that has run is priced at the average of its last three runs; otherwise at the most a run could cost (`upTo: true`): a crawl's pages at its highest tier's price, a rendered results page (4 credits; 1 when Bing answers), an agent run's budget. `basis` says how, in words. Free, and nothing is saved.

### `POST /playground/extract` — one URL with every format, for the app

`{url, allow_browser, use_proxy, config}` → an id; `GET /playground/{id}` for the result and its evidence; `…/screenshot` for the image.

## Projects

| Route | What |
|---|---|
| `GET /projects` · `POST /projects` `{seed, name?, schedule?, retention?, config?}` · `POST /projects/batch` `{seeds[], template_project_id?, schedule?, retention?}` | List; create one (`schedule` defaults to `manual` here — the app sends `weekly`); create many from a seed list with a template's config |
| `GET /projects/{id}` · `PATCH /projects/{id}` `{name?, schedule?, retention?, config?, paused?}` · `DELETE /projects/{id}` (admin) | Read; change anything — `paused: true` stops its scheduled runs and keeps the schedule (`nextRunAt` is null meanwhile; resumed, a slot that passed runs once; Run now still runs); delete with its runs and pages |
| `POST /projects/{id}/config/apply` `{project_ids[], groups[]}` | Copy this project's saved config onto others, by group (`scope`, `fetch`, `content`, `output`, …) |
| `POST /projects/run` `{project_ids[]}` | Start several |
| `POST /projects/{id}/dry-run` | Resolve the sources and sample URLs without spending |
| `GET /projects/{id}/sources` · `POST …/sources/preview` `{sitemap_include, sitemap_exclude, include_paths, …}` | The seed, sitemap, URL list, feeds and patterns with what the last run found through each; how many URLs a selection keeps |
| `GET /projects/{id}/discovery` · `POST …/discovery` · `GET …/discovery/entries?file=&cursor=` | The stored sitemap tree (files, sections, counts, `lastmod` ranges, the diff against the last snapshot); take a new snapshot (a map, billed); page through a file's entries |
| `POST /projects/{id}/selectors/match` `{selectors[]}` · `POST …/content/preview` `{config}` | Match counts against the last run's stored HTML; the markdown a config would produce for the seed |

## Runs and pages

| Route | What |
|---|---|
| `GET /projects/{id}/runs?limit=&cursor=&v=` · `POST /projects/{id}/runs` `{trigger?}` · `GET …/runs/{run}` · `POST …/runs/{run}/cancel` | History (25 a page); start (202); one run with its counts, credits, stop reason, engine breakdown, log; cancel |
| `GET /projects/{id}/pages?run_id=&v=&flag=&cursor=&limit=` | Rows of a run (100 a page, 1,000 at most). `v` filters by verdict; `flag=orphan\|unlisted\|dead_end`. The app's query language (`status:403 depth:>2`) is applied on the rows |
| `GET /projects/{id}/pages/content?url=&run_id=&format=` | One page's bodies, head, fields, versions |
| `GET /projects/{id}/pages/screenshot?url=&run_id=` | The JPEG |
| `POST /projects/{id}/pages/search` `{q, mode: content\|selector, run_id?}` | Which pages say this (words, "phrases") or contain this (CSS / XPath), scanned on the server |
| `POST /projects/{id}/pages/recrawl` `{urls[]}` | A scoped run of just these, now |
| `GET /projects/{id}/pages/removed?run_id=` | Pages in an earlier run and not this one |
| `GET /projects/{id}/runs/{run}/links?kind=orphans\|unlisted\|dead_ends` | The link graph's findings |

## Changes

| Route | What |
|---|---|
| `GET /projects/{id}/changes?run_id=` | The change record: `{counts: {added, removed, modified, same, withheld, unverified, skipped, fields, selectorBreaks, volatileBlocks}, coverage, baseline, entries[]}` |
| `POST /projects/{id}/changes/{change}/review` `{reviewed: true}` | Mark reviewed |
| `GET /projects/{id}/changes/page?url=&run_id=` | The word-level content diff of one page against the run before |
| `GET /projects/{id}/changes/markup?url=&run_id=` | The markup comparison: DOM diff, noise filter, selector repointing |

## Exports

`GET /exports` lists what can be exported. `GET /projects/{id}/export?dataset=pages|markdown|changes|fields|sitemap&format=jsonl|csv&run_id=` streams a dataset; `POST /projects/{id}/export` `{dataset, format, urls[]}` restricts it to listed URLs.

Two more datasets are text files, for a model rather than a table: `dataset=llms` is an [llms.txt](https://llmstxt.org) — the run's usable pages (`ok`, `thin`) as links with their descriptions, grouped by first path segment, under the site's title and the home page's description — and `dataset=llms-full` is the `llms-full.txt`: every usable page's markdown in one file, each under its title and a `Source:` line. Both take `format=txt` (the default for them) and the same `urls[]` selection.

## Connections and destinations

| Route | What |
|---|---|
| `GET /connections/providers` | The catalogue with each provider's fields, secrets, note and `verified` |
| `GET /connections` · `POST /connections` `{provider, name, config, secrets}` · `GET/PATCH/DELETE /connections/{id}` · `POST /connections/{id}/test` | Workspace connections; secrets are never returned |
| `GET /destinations` · `POST /destinations` · `PATCH/DELETE /destinations/{id}` · `GET …/syncs` · `POST …/sync` | Shared destinations (scope `all` or `projects`) and their syncs |
| `GET /projects/{id}/destinations` · `POST …` · `PATCH/DELETE …/{dest}` · `GET …/{dest}/syncs` · `POST …/{dest}/sync` | A project's own destinations |
| `POST /destinations/preview` `{project_id, shape, columns}` · `GET /destinations/shape-functions` | The shaped rows of the last run; the function and operator lists |

What goes in a destination, and how a shape is written, is on the [Connectors and destinations](/docs/connectors-destinations) page. A destination is written to a warehouse, database, storage, index or stream connection: one on a model (`llm`) or a [data provider](#data-providers) is a 400.

## Webhooks

**Project webhooks** — `webhook_url`, `webhook_events` and `webhook_include` in the config; `GET /projects/{id}/webhooks` (the events, recent deliveries, and the signing secret — to an admin or owner; anyone else gets `secretHidden: true`), `POST …/webhooks/test` (a `run.finished` with `test: true`), `POST …/webhooks/secret` (rotate, admin).

**Subscriptions** — a project's events sent to one more URL, beside its own webhook and without touching it: how an app such as Zapier or n8n turns them into a trigger, subscribing when a zap or workflow is switched on and deleting the subscription when it is switched off. `POST /webhooks/subscriptions` `{projectId, url, events?, label?}` (`member`) → 201 `{subscription: {id, projectId, url, events, label, createdAt, secret}, created: true}`. `events` are the project events below (`page.changed` when left out); `label` names who holds it in the list. Each subscription has a secret of its own, returned here only, and its messages are signed with it exactly as the project's own are. The same URL on the same project again returns that subscription (200, `created: false`) with the events now named. 400 on a URL that is not public http(s) or an event the project does not send; 409 `subscription_limit` past 25 on one project. `GET /webhooks/subscriptions?projectId=` lists the workspace's, or one project's, newest first and never with their secrets; `DELETE /webhooks/subscriptions/{id}` (`member`) → 204 stops it, dropping its messages still waiting. A receiver that answers 410 Gone is taken as having removed the hook: the subscription is deleted. Deleting the project deletes its subscriptions.

**Job webhooks** — the `webhook` on `POST /crawl` and `webhook_url` on a batch `/scrape`, with their own secret returned once. Events `crawl.started`, `crawl.page` (gathered into one message per 50 pages, or every 5 s), `crawl.completed`, `crawl.failed`, `scrape.completed`, `scrape.failed`, `map.completed`, `batch.finished`; `metadata` comes back on every message. An agent run's `webhook` (`POST /agent`) has its own five: `agent.started`, `agent.action`, `agent.completed`, `agent.failed`, `agent.cancelled`. A search's `webhook` (`POST /search`) is told once how it ended: `search.completed` or `search.failed`. A monitor's `webhook` (`POST /monitors`) hears `search.changed` or `agent.changed`, once a run that changed something, `budget.reached` once a month when its monthly cap skips a run, and — an agent monitor that follows its answer as `rows` — `row.changed` with the records that moved.

**Page markdown in the message** — off unless asked. `"webhook_include": ["markdown"]` in a project's config adds each added or modified page's `markdown` to its `page.changed`; `"include": ["markdown"]` in a crawl's `webhook` (`{"url": …, "events": ["crawl.page"], "include": ["markdown"]}`) adds each page's to `crawl.page`. A message carries up to about 1 MB of markdown; a page past that, or one not stored, has `markdown: null`, `markdownOmitted` (`too_large` or `not_stored`) and a `contentUrl` — the API path that returns it in full (`/projects/{id}/pages/content?url=&run_id=`, or `/crawl/{id}/page?url=`), called with your key. A removed page has none. Subscriptions are sent the message without it. The delivery log (`GET /webhooks/deliveries`) shows `[sent in full]` in place of each page's markdown; the receiver is sent it whole, signed as every message is. A crawl's `crawl.page` deliveries that carried markdown are kept 7 days (a one-shot crawl that is not kept takes its deliveries with it when it expires).

**Project events** — `run.finished` (status, pages, counts, coverage, stop), `page.changed` (every page added / modified / removed with words added and removed, the section, whether it was volatile; a modified page only when its edit reaches `notify_min_words`), `page.blocked`, `field.canonical` (canonical, robots and title changes), `site.failing` / `site.recovered`, `budget.reached` (a scheduled run skipped because the project's `max_credits_per_month` is spent; once a month), `page.orphaned` / `page.linked`, `sitemap.added / removed / updated / file_added / file_removed`.

**The digest** — with `change_digest: {connection_id, focus}` in the config (an `llm` connection of the workspace; `focus` is what the reader cares about, e.g. "pricing and plan limits"), a compared run with something in it asks the model for a paragraph on what changed and why it matters, up to five points and a `significant` flag. It rides on `run.finished` as `data.digest` (`{summary, points[], significant}`, or `null`), in the run email, in the chat message, and on the change set (`GET /projects/{id}/changes` → `digest`). Tokens are charged like enrichment, through your own key.

**Chat channels** — `chat_url` in the config is a Slack, Discord, Microsoft Teams (Workflows), Mattermost or Google Chat incoming-webhook URL. When a run finishes with something to say — the same rule as the run email: never for a run that compared clean — the record is posted there in that service's own shape (Block Kit, an embed, an Adaptive Card, or `{text}`), unsigned, through the same delivery queue and retries as a webhook; `site.failing` / `site.recovered` post too. The URL is a secret to the channel, so the API returns it masked (`https://hooks.slack.com/services/••••`), and a masked value sent back in a `PATCH` keeps the one stored; send `""` to remove it. `GET /webhooks` lists the channel with its health beside the project's webhook (`kind: "chat"`, `service`).

**The message** — `POST`, JSON `{id, event, createdAt, data}`, headers `X-MeshArc-Event`, `X-MeshArc-Delivery`, `X-MeshArc-Signature: sha256=<HMAC-SHA256 of the raw body with the secret>`, `User-Agent: MeshArc-Webhooks/1.0`. Answer 2xx within 10 s.

**Retries** — no answer, a 5xx, a 408 or a 429 is retried six times over about a day (1 m, 5 m, 30 m, 2 h, 6 h, 12 h), then the delivery is `paused`. Any other 4xx is your endpoint's answer — a wrong URL, a rejected signature — and the delivery is `failed` at once. `GET /webhooks/deliveries?runId=|projectId=&limit=` lists what was sent, attempt by attempt (`status`, `attempts`, `lastStatus`, `lastError`, `nextAt`, the log); `POST /webhooks/deliveries/{id}/redeliver` sends one again now with its attempts kept. `GET /webhooks` gives every project's webhook with its health.

## The workspace

| Route | What |
|---|---|
| `GET /me` · `PATCH /me` `{max_rpm}` (admin) | The workspace, its plan and limits, credits used and remaining, what this key may do; the rate ceiling |
| `GET /me/usage` · `GET /me/monitor` | Pages per day and this month's pages, runs, blocked share, pages by engine; what is queued and running, lane depths |
| `GET /me/keys` · `POST /me/keys` `{name, scopes[], projects[], expires_in_days, rpm}` (admin) · `DELETE /me/keys/{id}` | Keys; the secret is in the create response only. Leave `projects` out for a key to the whole workspace; an empty list is refused (400), never read as that. A key's `rpm` only lowers the plan's, and is shown as the limit it meets |
| `GET /me/billing` · `POST /me/billing/subscribe` · `/change` · `/confirm` · `/card` · `/cancel` · `/keep` · `/credits/buy` · `PATCH /me/billing/cap` · `GET /me/billing/invoices` · `/receipts` · `/credit-notes/{id}` | The plan, the meter, the ledger, the price list; subscribing, changing plan and card through Razorpay Checkout (`confirm` verifies Checkout's answer), cancelling and undoing it, buying credits, the overage cap; invoices, receipts and credit notes |
| `GET /me/notifications` · `POST /me/notifications/{id}/seen` | Signed-in members only: while the server has no mail host (`emailConfigured: false`), the run records and monitor alerts email would have carried — `{id, kind, subject, text, link, createdAt}`, unseen, a week at most — and dismissing one. Empty once mail is set up, and for a key |
| `GET /orgs/children` · `POST /orgs/children` `{name, monthlyCap, externalId}` · `GET /orgs/children/{id}` · `PATCH /orgs/children/{id}` `{name, monthlyCap, suspended, reason}` · `GET/POST /orgs/children/{id}/keys` · `DELETE /orgs/children/{id}/keys/{key}` (admin) | [Sub-accounts](#sub-accounts): made, capped, suspended, given keys; one in full is its month, its spend by kind, its latest charges and which of your keys acted on it |
| `GET /me/proxies` | The pool with per-exit parity and success |
| `GET /meta` | Verdict meanings, engine costs, the config defaults, the ladder |
| `/auth/*` | `POST /auth/signup`, `/login`, `/logout`, `GET /session`, `POST /switch`, `/verify/request`, `/verify`, `/forgot`, `/reset`, `PATCH /profile`, `POST /password`, `GET/DELETE /sessions`, `GET /members`, `POST /invites`, `DELETE /invites/{id}`, `GET /invites/{token}`, `POST /invites/accept`, `PATCH/DELETE /members/{user}`, `PATCH /workspace`, `GET /sso`, `POST /sso/{provider}/start`, `POST /sso/{provider}/finish`, `GET /identities`, `DELETE /identities/{provider}` |

## Sub-accounts

For a platform that runs MeshArc for its own customers: a workspace of its own for each one, made by yours. A sub-account has its own keys, projects, runs, monitors and data, and sees nothing of your workspace or of its siblings; your workspace sees its spend and can act on it.

- **Plans.** Sub-accounts come with Scale (25) and Enterprise (500, or what was agreed); on another plan `POST /orgs/children` is `402 plan_limit`. A sub-account is on your plan — whatever your plan is now — with its limits, and has the plan's concurrency of its own. It cannot have sub-accounts. Moving to a plan without them keeps the ones you have; it stops you making more.
- **Credits.** A sub-account has no credits of its own: it spends yours, and what it spends is on your bill — your month's `counters.credits` includes it, and `counters.pool` says how much of it was your sub-accounts'. Give one a `monthlyCap` and it stops at that many credits a month, whatever you have left; with none (`0`) it may spend what you have. Run out, or reach its cap, and the next job it starts is refused with `402`, saying which; what it has already stored stays readable. Its own `GET /me/billing` says who pays (`month.billedTo`) and refuses the billing actions — plan, card, credits, overage cap — with `409 sub_account`.
- **Keys.** `POST /orgs/children/{id}/keys` makes it a key, shown once, for your customer: it reaches that sub-account and nothing else. Or act on it with one of your own keys and the `X-MeshArc-Tenant` header ([Authentication](#authentication)).
- **Suspending.** `PATCH {"suspended": true, "reason": "…"}` refuses its keys and your header, and cancels what it has queued or running; `false` lifts it. One that MeshArc suspended, or whose parent is suspended, is `stopped`: not yours to lift.
- **Deleting.** Not offered: suspend one instead. Its data follows the retention of its projects, as any workspace's does.

```bash
# Make one for a customer, at most 50,000 credits a month
curl -X POST https://api.mesharc.dev/api/v1/orgs/children \
  -H "Authorization: Bearer $MESHARC_KEY" -H "Idempotency-Key: cust-42" \
  -d '{"name": "Acme Ltd", "monthlyCap": 50000, "externalId": "cust-42"}'

# Scrape on its behalf, with your own key
curl -X POST https://api.mesharc.dev/api/v1/scrape \
  -H "Authorization: Bearer $MESHARC_KEY" -H "X-MeshArc-Tenant: <its id>" \
  -d '{"urls": ["https://example.com"]}'
```

## Saved logins

A workspace's signed-in session on a site, saved from a live browser, used by the projects and scrapes that name it (`login_id` in their `config`) and by agent runs, which sign in with one where a page on its site needs the account (`logins` on `POST /agent` and `POST /agent/batch` says which they may use). Switched on per deployment: while it is off, `GET /logins` says so (`enabled: false`), saving or testing one answers `409 logins_off`, and a project, scrape or agent run that names one is refused (400). Owners and admins only.

| Route | What |
|---|---|
| `GET /logins` | `{data, enabled, why}`: each login's `id`, `host`, `name`, `status` (`active` or `expired`), how many cookies it holds, who saved it and when, the `terms` they accepted, `uses` and `lastUsedAt`, and why it expired. Never the session |
| `PATCH /logins/{id}` `{name, expired}` | Rename it; `expired: true` stops using it now |
| `DELETE /logins/{id}` | Delete it. Projects and runs that name it stop, saying so; none reads the site signed out instead |
| `POST /logins/{id}/test` | One plain fetch of the site's front page with the login: `{status, finalUrl, loginPage, reason}`. Free |
| `GET /logins/terms` | The terms a person accepts before saving one, and whether logins are on |
| `POST /logins/live` `{host, name, acknowledged, renew?}` | For a person signed in to the app (not a key): a single-use `ticket`, good for a minute, and the `wsUrl` of the live browser's socket. `renew` is the id of the login the sign-in saves again (it must be this workspace's, for that site) |

**How a login is used.** Only on its host and that host's subdomains: its cookies go where a browser would send them — one the site set for its own host stays on that host, never sent to a subdomain or across a redirect to another host — and a page rendered with it gets a browser of its own. A login is for a site's name: an IP address, or a name many unrelated sites share (`co.uk`, `github.io`), is refused. A run that meets the site's login page with it marks it expired (`status: expired`, `expiredReason`) and emails the workspace's owners and admins once; runs that name it then fail until it is saved again — and signing in again to that login (**Sign in again**, or `renew` with its id) renews it, under the same `id`, so the projects, monitors and runs that name it work again. Without `renew`, a save renews the site's expired login only when the site has no other, or when it is given that login's exact name; otherwise it is a new login, so one account's session is never saved under another's. A key held to some projects cannot list or change the workspace's logins (`403 project_key`).

**The live browser's socket.** Open `wsUrl` from the app's own pages. The server sends `{"type": "frame", data, width, height}` (a JPEG of the page, base64), `{"type": "page", url, title}` after each step, `{"type": "saved", login}`, `{"type": "error", message}` and `{"type": "closed", reason}`. Send `{"type": "click" | "dblclick", x, y}` (in the frame's pixels), `{"type": "text", text}`, `{"type": "key", key}` (`Enter`, `Tab`, `Shift+Tab`, `Backspace`, `Escape`, the arrows, `Home`, `End`, `PageUp`, `PageDown`), `{"type": "scroll", dy}`, `{"type": "navigate", url}`, `{"type": "back"}`, `{"type": "reload"}`, then `{"type": "save", name}` once signed in — or `{"type": "close"}`. A save keeps the site's own cookies and local storage, encrypted; the password typed into the site's form is never kept. The browser opens only public sites: an address on this deployment's own network — typed, linked or loaded by a page, a page's WebSocket included — is never reached, and every connection is judged on the address it is made to, so a name cannot answer one address to the check and another to the browser. The browser closes after a few minutes, or sooner with no input; one per person.

## Data providers

Company and financial data the agent looks up on the workspace's own key, from two providers. Connect one as any connection is (`POST /connections`, `{"provider": "pdl" | "fmp", "secrets": {"api_key": "…"}}`; owners and admins), then name it on a run with `dataConnections`: `POST /agent`, a follow-up, `POST /agent/batch`, or an agent monitor's `request`. The run is given that provider's tool; a run naming none has neither.

| Provider | The agent's tool | What it looks up |
|---|---|---|
| **People Data Labs** (`pdl`) | `company_data` | One company, matched by its website (best), LinkedIn page or ticker, or loosely by name and country: name, size and employee count, industry, type, founding year, headquarters, LinkedIn, summary and tags, funding raised, last round and its date, inferred revenue. With PDL's `likelihood` (1–10) of the match |
| **Financial Modeling Prep** (`fmp`) | `financial_data` | A listed company: `search` (a name → its ticker symbols), `profile`, `quote`, and `income`, `balance`, `cashflow` and `metrics` (annual or quarterly, newest first, up to 5) |

- **What it costs:** no MeshArc credits. The provider counts each lookup on your plan — People Data Labs bills a match (a lookup that matches nothing is free), and Financial Modeling Prep counts a call. A run makes at most 25 lookups, and the same lookup twice in a run is one call.
- **Where a value came from:** each record found is one of the run's `sources`, `{url, title, provider}` — `url` is the request it came from (`https://api.peopledatalabs.com/v5/company/enrich?website=stripe.com`), for `fieldSources` to point to. Your key is sent in a header and is never in a source, the trace, the answer or a webhook.
- **In the trace:** each lookup is a `lookup` step with the provider, what was looked up, `found`, `likelihood` (PDL) or the `error`, and 0 credits.
- **When it cannot answer:** a refused or spent key, a plan that does not include the data, or the provider limiting requests is the tool's answer, said plainly; the run goes on with what it has. A connection deleted after a run was asked for leaves that run without its tool, and its trace says so. A 429 is waited out once (up to 10 s).
- **Testing a connection:** `POST /connections/{id}/test` makes one request: People Data Labs' free Autocomplete, or one Financial Modeling Prep symbol search (one of your plan's calls).

Both are built to the providers' current documentation and marked unverified until run against a live account.

## What a page row says

| Field | Values |
|---|---|
| `verdict` | `ok`, `thin` (short but a page; kept), `blocked` (refused, or a 404 / soft 404 — see `errorCode`), `skipped` (under the word floor or outside the language list; the row stays, out of the change record), `robots`, `cached` |
| `errorCode` | `OK`, `SHELL`, `BLOCKED`, `NOT_FOUND`, `RATE_LIMITED`, `CAPTCHA`, `LOGIN_REQUIRED`, `GEO_RESTRICTED`, `TIER_LIMIT` (a higher tier than allowed would have read it), `BUDGET_EXHAUSTED` (the credits the workspace had left could not buy the engine that would have read it), `TIMEOUT`, `NETWORK`. A short page is `OK` with a warning |
| `shape` | `listing`, `table`, `form`, or empty — what a short page's markup says it is |
| `warnings` | `short` |
| `signals` | Why the judge decided as it did: `nav`, `30 links`, `json-ld`, `open graph`, `main`, `no script`, `app shell`, `listing`, `render confirmed`, `soft 404`, `host fingerprint`, `soft404?` |
| `method`, `tier`, `climbedTo` | The engine that answered and its tier; the highest engine a refused page was tried on |
| `credits`, `billedAs` | What the page cost; the rung it is priced at when not the one that fetched it |
| `widget`, `solved`, `exitBytes` | A captcha widget on the page and what was done; bytes through the residential exit |
| `foundVia`, `firstReferer`, `inSitemap`, `inLinks`, `orphan`, `unlisted` | How the crawl reached it and what the link graph says |

---

# SDKs and MCP

Source: https://mesharc.dev/docs/sdks

# SDKs and the MCP server

Both SDKs are thin: every method is one API call and returns the API's JSON unwrapped, so the [API reference](/docs/api) is the reference for them too. Both read `MESHARC_API_KEY` from the environment.

Building with LangChain? [`langchain-mesharc`](/docs/langchain) wraps the Python SDK as a document loader and agent tools.

## Python

Python 3.10+, one dependency (`httpx`), fully typed.

```python
from mesharc import MeshArc, MeshArcError

arc = MeshArc("mesharc_...")                     # or MeshArc() with MESHARC_API_KEY set

# One URL, waited for: the API holds the request open for the page
page = arc.scrape("https://quotes.toscrape.com/js/", config={"render_js": "always"})
print(page["markdown"][:200], page["credits"], page["verdict"])

# Many URLs, no project: grouped by host, fetched in parallel where allowed
batch = arc.scrape(["https://a.com/", "https://b.com/x"], config={"concurrency": 4})
for row in batch["pages"]:
    print(row["url"], row["httpStatus"], row["verdict"], row["credits"])

# What a site declares, before fetching any of it
for u in arc.map("https://docs.example.com"):
    print(u["url"], u["lastmod"])

# A whole site, once, with no project to set up first
job = arc.crawl("https://docs.example.com", limit=200, maxDepth=3,
                scrapeOptions={"formats": ["markdown", "links"]})
for page in job.pages():                         # follows the cursor, keeps up with the crawl
    print(page["url"], page["words"])
print(job.status, job.envelope["counts"], job.envelope["creditsUsed"])
project = job.keep(name="Docs", schedule="weekly")   # ...and keep it, if it is worth watching

# A site watched over time
project = arc.projects.create("https://docs.example.com", name="Docs", schedule="weekly",
                              config={"max_pages": 300, "include_paths": ["/docs/*"]})
run = arc.runs.start(project["id"], wait=True)
record = arc.changes(project["id"])
print(record["change"]["counts"])
hits = arc.search(project["id"], '"free shipping" returns')             # words in the markdown
hits = arc.search(project["id"], "table.pricing td", mode="selector")   # elements in the html
arc.export(project["id"], "pages.jsonl", dataset="markdown")
```

| Method | Calls |
|---|---|
| `extract(url, config, wait, poll, timeout)` | `POST /playground/extract` — one URL with every format |
| `scrape(urls, config, webhook_url, formats, wait, …)` · `scrape_one(url, config, formats, wait, timeout_s, idempotency_key)` | `POST /scrape` |
| `crawl(url, wait, poll, timeout, idempotency_key, **opts)` → `Crawl` (`.status`, `.envelope`, `.refresh()`, `.wait()`, `.pages(cursor=…)`, `.page(url)`, `.keep()`, `.cancel()`) · `get_crawl(id)` | `POST /crawl`, `GET /crawl/{id}`, `…/page`, `…/keep`, `DELETE` |
| `map(url, **opts)` → rows · `map_details(url, search, limit, …)` → the envelope | `POST /map` |
| `batch(batch_id, formats, wait, …)` | `GET /scrape/{id}` |
| `projects.list() / create(seed, name, schedule, retention, config) / get / update(project_id, **fields) / delete` | `/projects` |
| `runs.list(project_id, limit) / get / start(project_id, wait, poll, timeout) / wait / cancel` | `/projects/{id}/runs` |
| `pages(project_id, run_id)` · `page(project_id, url, run_id)` · `changes(project_id, run_id)` · `page_diff(project_id, url, run_id)` · `search(project_id, q, mode, run_id)` · `recrawl(project_id, urls)` · `sources(project_id)` · `export(project_id, path, dataset, fmt, run_id, urls)` | The project routes |
| `me()` · `keys()` · `create_key(name, scopes, projects, expires_in_days, rpm)` · `revoke_key(key_id)` · `usage()` · `monitor()` · `meta()` | The workspace routes |
| `close()` | Close the HTTP client |

Errors raise `MeshArcError` with `status`, `detail`, `code` (`validation`, `rate_limited`, `plan_limit`, `not_found`, …) and `request_id` — the id the API put on the response header and in its logs. A job the client stopped waiting for raises `MeshArcTimeoutError` (also a `TimeoutError`) with its `job_id`. Requests time out after `timeout` seconds (150) and are retried on 429, 502, 503, 504 and network failures when safe to repeat — a GET, a DELETE, or a POST with an `idempotency_key=` — up to `max_retries` times (2).

## Node

Node 20+, zero dependencies (native `fetch`). TypeScript types, ESM and CommonJS builds.

```ts
import { MeshArc, MeshArcError } from 'mesharc';

const arc = new MeshArc('mesharc_...');            // or new MeshArc() with MESHARC_API_KEY set

const page = await arc.scrape('https://quotes.toscrape.com/js/', { render_js: 'always' });
console.log((page.markdown as string).slice(0, 200), page.credits, page.verdict);

const batch = await arc.scrape(['https://a.com/', 'https://b.com/x'], { concurrency: 4 });
for (const row of batch.pages) console.log(row.url, row.httpStatus, row.verdict, row.credits);

for (const u of await arc.map('https://docs.example.com')) console.log(u.url, u.lastmod);

const job = await arc.crawl('https://docs.example.com', { limit: 200, maxDepth: 3 });
for await (const p of job.pages()) console.log(p.url, p.words);       // follows the cursor
const kept = await job.keep({ name: 'Docs', schedule: 'weekly' });

const project = await arc.projects.create('https://docs.example.com', { name: 'Docs', schedule: 'weekly' });
const run = await arc.runs.start(project.id, { wait: true });
const record = await arc.changes(project.id);
const hits = await arc.search(project.id, 'table.pricing td', 'selector');
const csv = await (await arc.export(project.id, { dataset: 'pages', format: 'csv' })).text();
```

| Method | Calls |
|---|---|
| `extract(url, config?, {wait, poll, timeout})` | `POST /playground/extract` |
| `scrape(urls, config?, {webhookUrl, formats, apiTimeoutS, idempotencyKey, …})` · `scrapeOne(url, config?, {…})` | `POST /scrape` |
| `crawl(url, {limit, maxDepth, …, wait, idempotencyKey})` → `Crawl` (`.status`, `.envelope`, `.refresh()`, `.wait()`, `.pages()` async iterator, `.keep()`, `.cancel()`) · `getCrawl(id)` | `/crawl` |
| `map(url, {search, limit})` → rows · `mapDetails(url, {…})` → the envelope | `POST /map` |
| `batch(batchId, {formats, wait})` | `GET /scrape/{id}` |
| `projects.list() / create(seed, {name, schedule, retention, config}) / get / update(id, fields) / delete` · `runs.list / get / start(id, {wait}) / wait / cancel` | `/projects` |
| `pages`, `page`, `changes`, `pageDiff`, `search(projectId, q, 'content'|'selector', runId?)`, `recrawl`, `sources`, `export(projectId, {dataset, format, runId, urls})` → `Response` | The project routes |
| `me()`, `keys()`, `createKey(name, {scopes, projects, expiresInDays, rpm})`, `revokeKey`, `usage()`, `monitor()`, `meta()` | The workspace routes |
| `call(method, path, body?, params?)` · `raw(...)` | Any route |

Errors throw `MeshArcError` with `status`, `detail`, `code` and `requestId`; a job the client stopped waiting for throws `MeshArcTimeoutError` with its `jobId`. Requests time out after `timeoutMs` (150 000) and are retried on 429, 502, 503, 504 and network failures when safe to repeat — a GET, a DELETE, or a POST with an `idempotencyKey` — up to `maxRetries` times (2).

## The MCP server

The Python package also ships MeshArc as an MCP server, so Claude Desktop, Claude Code, Cursor and any MCP client can scrape, crawl, map, set up and change projects, and read change records as tools. There is a hosted one at `https://mcp.mesharc.dev/mcp` -- paste the URL, sign in, approve, and nothing is installed -- or you can run it yourself over stdio with your own key. Either way it sits on top of the Python client, so what an agent gets is what the API gives. A result with many pages in it is an index of the pages read (up to 500, fewer if the budget needs the room), an excerpt of as many as a 60,000-character budget pays for (fifty at most), and a count of each page’s links rather than the links; ask for one page on its own, each body up to 12,000 characters, with `get_job(kind="crawl", id=…, url=…)`, or carry on through a crawl with `cursor=`. Hosted, a connection approved read-only can read what the workspace has stored — projects, pages, change records, search, `get_job` — but not fetch a new page or change anything: those tools reach the site or alter the workspace, and most of them spend credits; those nine tools answer `read_only` until the app is reconnected with write access. Python 3.10+. The [MCP page](/mcp) walks through setup for each client and shows real conversations.

```bash
pip install "mesharc[mcp]"
MESHARC_API_KEY=mesharc_... mesharc-mcp

# Claude Code
claude mcp add mesharc -e MESHARC_API_KEY=mesharc_... -- mesharc-mcp
```

| Tool | What it does |
|---|---|
| `scrape_urls(urls, config?, formats?, parse_documents?, full?)` | Content for a list of pages; PDFs and Word (`.docx`) files read as text unless told not to. `full=true` reads one page with every format |
| `map_site(url, search?, limit?)` | The site's URLs from its sitemap |
| `search_web(query, limit?, country?, lang?, freshness?, include_domains?, exclude_domains?, scrape?)` | Search the web: ranked results, kept to named sites when asked, each page as markdown with `scrape=true` |
| `crawl_site(url, limit?, max_depth?, include_paths?, …)` | A one-shot crawl, pages returned as they land; the paths are globs (`/blog/*`) |
| `get_project()` | With no project id: every setting an assistant can set, its meaning and default — read before setting one |
| `list_projects()` · `get_project(project_id)` · `create_project(seed or crawl_id, name?, schedule?, config?)` · `update_project(project_id, name?, schedule?, config?, test_webhook?)` | Projects and their settings; `crawl_id` promotes a one-off crawl, a partial `config` on update merges, and a key that is not a setting is refused by name. `get_project` shows the webhook and its last deliveries; `test_webhook=true` sends it a test; `list_projects` also says the credits left |
| `delete_project(project_id, confirm_name)` | Delete a project and its history; needs an admin API key, so only the local server can |
| `start_run(project_id, urls?, wait?)` · `list_runs(project_id, limit?)` | Run a project now, or only the pages listed; its runs, newest first |
| `list_pages(project_id, run_id?, q?, mode?, url?)` | A run's pages; with `q` a search of them; with `url` one page in full, with its versions |
| `get_changes(project_id, run_id?, against?)` | The change record, against the run before or any run named with `against` |
| `get_job(kind, id, …)` · `cancel_job(kind, id, project_id?)` | Follow a long job; stop one |

The clients are open source: [mesharc-python](https://github.com/mesharc-org/mesharc-python) and [mesharc-node](https://github.com/mesharc-org/mesharc-node).

---

# LangChain

Source: https://mesharc.dev/docs/langchain

# LangChain

`langchain-mesharc` puts MeshArc inside LangChain: a document loader that turns web pages into `Document`s, and two tools an agent can call to search the web and read pages. Its distinctive mode is `changes`, which loads only the pages that changed since a project's last crawl, so a vector store re-embeds what changed and nothing else.

It is listed in LangChain's [document loader integrations](https://docs.langchain.com/oss/python/integrations/document_loaders), and the source is [mesharc-org/langchain-mesharc](https://github.com/mesharc-org/langchain-mesharc) (MIT).

## Install

```bash
pip install langchain-mesharc
```

Python 3.10 or newer, with `langchain-core` 1.x. It uses the [Python SDK](/docs/sdks) underneath.

Make an API key under **Settings → API keys** in the app and set it as `MESHARC_API_KEY`, or pass `api_key=` to the loader or a tool. Reading, crawling, mapping and searching need a key with **write** scope; `changes` works with a read-only key. The free plan includes 1,000 credits, with no card.

## The document loader

```python
from langchain_mesharc import MeshArcLoader

# One page, or several read as one batch
docs = MeshArcLoader("https://example.com/pricing").load()
docs = MeshArcLoader(["https://example.com/", "https://example.com/about"]).load()

# A whole site, or a section of it; pages arrive as they are read
loader = MeshArcLoader("https://docs.example.com", mode="crawl",
                       params={"limit": 100, "includePaths": ["/guides/*"]})
for doc in loader.lazy_load():
    print(doc.metadata["url"], len(doc.page_content))

# The URLs a site declares in its sitemaps, without fetching the pages
urls = MeshArcLoader("https://docs.example.com", mode="map").load()

# What changed on a watched site in its latest run
changed = MeshArcLoader(mode="changes", project_id="<project id>").load()
```

| Mode | Input | One `Document` per |
|---|---|---|
| `scrape` (default) | a URL or a list of URLs | page read, holding its markdown |
| `crawl` | a site's URL; `params` takes the [crawl options](/docs/api), such as `limit`, `maxDepth`, `includePaths` and `excludePaths` | page read in the crawl |
| `map` | a site's URL; `params` takes `search` and `limit` | URL in the site's sitemaps, with the URL as its text |
| `changes` | `project_id`; optionally `run_id` (the latest finished run by default) and `against` (compare with this run instead of the one before) | page added, modified or with changed fields, holding the page as it is now |

`params` go to the SDK call unchanged, so any option the API takes for that call works here, by its API name.

**Pages with no content are left out.** A page that was blocked or returned a 404 yields no `Document` unless you pass `include_empty=True`; its `page_status` and `error_code` then say why. In `changes` mode, removed pages are left out unless you pass `include_removed=True`, and they arrive with empty text.

## Metadata

Every value is a string, a number or a boolean, so any vector store accepts it.

| Key | On | What it is |
|---|---|---|
| `source`, `url` | every page | the page's URL |
| `title` | every page | the page title |
| `page_status` | every page | one word: `ok`, `missing`, `blocked`, `captcha` and so on |
| `crawled_at`, `mode` | every page | when it was read, and which mode loaded it |
| `status` | `scrape`, `changes` | the HTTP status (crawl results don't carry it) |
| `verdict`, `credits` | `scrape` | how the read was judged, and what it cost |
| `crawl_id` | `crawl` | the crawl the page came from |
| `project_id`, `run_id`, `previous_run_id`, `changed_at` | `changes` | which run, compared with which |
| `change` | `changes` | `added`, `modified`, `removed` or `fields` |
| `words_before`, `words_after` | `changes` | the page's length either side of the change |

In `changes` mode each `Document`'s `id` is `<run id>:<url>`.

## Keeping a vector store current

Point a [project](/docs/app) at the site with a schedule, then after each run load only what changed. Keying each page by its URL makes re-adding a page replace the old copy, and `include_removed=True` tells you what to delete.

```python
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_mesharc import MeshArcLoader

store = InMemoryVectorStore(embeddings)   # any LangChain embeddings and vector store

def refresh(project_id: str) -> None:
    docs = MeshArcLoader(mode="changes", project_id=project_id, include_removed=True).load()
    gone = [d.metadata["url"] for d in docs if d.metadata["change"] == "removed"]
    fresh = [d for d in docs if d.metadata["change"] != "removed"]
    if gone:
        store.delete(ids=gone)
    if fresh:
        store.add_documents(fresh, ids=[d.metadata["url"] for d in fresh])
```

Run `refresh` from the project's `run.finished` [webhook](/docs/api), or on the same schedule as the project. A run that reached less than 90% of the site withholds its removals, so a blocked crawl never deletes pages from your index.

If you split pages into chunks before embedding, give each chunk an id that starts with the page's URL, and delete a page's old chunks before adding its new ones.

## Tools for agents

```python
from langchain_mesharc import MeshArcScrape, MeshArcSearch

tools = [MeshArcSearch(), MeshArcScrape()]
```

| Tool | Name | The agent gives | It gets back |
|---|---|---|---|
| `MeshArcSearch` | `mesharc_search` | `query`; `limit` (1 to 10, default 5); `read_pages` | a numbered list of results (title, URL, snippet), and with `read_pages` each result's page text |
| `MeshArcScrape` | `mesharc_scrape` | `urls` (1 to 20) | each page's title, URL and markdown, or why it could not be read |

Both return `content_and_artifact`: the content is the text the model sees, with each page cut at `max_chars_per_page` (8,000 by default; `0` for no limit), and the artifact is MeshArc's full response. A refused request, such as one with too few credits left, comes back to the agent as an error message rather than stopping the run.

## What it costs

Requests are billed in credits like any other API call: a plain page read is 1 credit, a rendered page 4, and blocked pages and 404s cost nothing. A web search costs what reading its results page costs, and a map costs 1 credit per sitemap file. `changes` reads pages the project already stored. See [Credits, plans and billing](/docs/billing).

---

# Credits, plans and billing

Source: https://mesharc.dev/docs/billing

# Credits, plans and billing

## How credits work

A credit is the unit of everything. **A page costs what it took to read**: the ladder starts cheap and climbs only when a site refuses, and you pay for the rung that produced what you got. You never choose the engine; you set a ceiling (the tier) and, if you want, a cap per page.

### Reading a page

| Mode | What happens | Credits |
|---|---|---|
| **Fetch** | A plain HTTP request with a real browser's fingerprint | **1** |
| **Fetch with session** | The same request carrying cookies a browser earned on this site earlier. One render earns the session; every page after it is read this way | **2** |
| **Render** | A real Chrome loads the page, runs its JavaScript, waits for it to settle | **4** |
| **Render, residential** | The same Chrome from a residential address; 1 MB of traffic included | **16**, +4 per further started MB |
| **Render, solved** | A render plus a paid answer for a captcha widget on a form your steps submit. A widget passed on a click is a render | **22** |
| **Unlocked** | A rented fetch through a specialist service; the last rung, stealth only | **40** |

### Add-ons and the other things

| | Credits |
|---|---|
| A screenshot | +1 |
| A PDF document | at least 2 (a fetched one 2, a rendered one 4) |
| Structured extraction with a model on your own key | 1 per 1,000 tokens in and out (the model bill is yours) |
| A map (`POST /map`, or **Discover** on the Sources screen) | 1 per sitemap file read, at the rung that read it — most sites 1; never per URL declared |
| A render bought on suspicion that found a short page unchanged | the http rung's price; the render is on the ledger at 0 |

### Web search and the agent

A search and an agent run are priced the way pages are: by what they read.

| | Credits |
|---|---|
| A web search (`POST /search`): one results page, at the rung that read it — typically a render for Google | **4** (**1** when Bing answers) |
| Each further page of results (`page` 2–10) | its own results page, priced the same |
| The same search again within the hour (ten minutes for news) — same query, country, language, freshness and sites | **0**, answered from the last one |
| A results page every engine refused | **0** |
| Each result's page, when the search asks to scrape them | what that page took to read, as any page |
| An agent run (`POST /agent`): the pages it reads | what each took to read; a refused page is free |
| The agent's model tokens, on MeshArc's model | the model provider's price **+ 20%**, in credits (Billing shows the rate per million tokens) |
| The agent's model tokens, on your own model connection (`connectionId`) | **1** per 1,000 tokens; your provider bills the tokens to your key |
| A page a conversation already read, read again in a follow-up or a continued run | **0** |
| The agent over a table (`POST /agent/batch`) | each row as one agent run, up to `maxCreditsPerRow`; the table never spends more than rows × that cap |
| Checking a table first (`POST /agent/batch/preview`) and its estimate | **0** |
| A monitor's run (a search or a question on a schedule) | what that search or agent run costs; a daily Google search is about 120 a month |

An agent run stops at its `maxCredits` (2,000 unless you set another) with what it found, and **Continue** carries on with a new budget. A monitor can have a monthly cap (`budget.perMonth`) and an agent monitor a cap per run (`budget.perRun`); `GET /estimate` says about what a month costs before you save one.

### Free, always

- A page the site **refused**, whatever the ladder spent finding out — you pay for pages, not for walls.
- A **404 or 410**, and a 200 that is the site's not-found page.
- A page **served from the cache** (`cache_max_age_hours`), an **incremental copy** carried forward when the sitemap says it did not change, a URL **robots.txt** kept us from.
- The discovery a run makes for itself, the change record, webhooks, exports, destinations.
- A document the project did not ask for (recorded as seen).
- A search results page every engine refused, and the same search again within the hour (ten minutes for news).
- Checking an agent table before it runs, and a page an agent conversation already read, read again.

### Tiers

The ceiling you set per project: `http` = Fetch and Fetch with session (every plan) · `browser` = + Render (Free, Starter and up) · `stealth` = + residential, solved, Unlocked (Growth and up) · `auto` = as high as your plan allows (a project made with **New project** starts here). A page still starts where the site is known to answer (the cheapest engine, on a site not read before) and climbs only when refused. The plan caps the tier on a project's runs, a scrape, the playground and a map alike. A page that needs more than the tier allows is recorded `TIER_LIMIT` at 0 — "this costs more than you allow", not "unreadable". `max_credits_per_page` (`maxCredits` on `/scrape`) is a finer cap. A project also takes `max_credits_per_run` and `max_credits_per_month`, and a monitor a `budget` (`perRun`, `perMonth`): a run stops at the first, and a scheduled run due once the month's is spent is skipped. `GET /estimate` says about what a schedule costs a month before you save it.

### In practice

| | Credits |
|---|---|
| A 200-page documentation site, plain fetch, read twice | 400 |
| The same site, rendered (JavaScript-only), read twice | 1,600 |
| A 100-page Cloudflare-protected site: one render earns the session, 99 pages read with it | 4 + 198 = **202** |
| A residential render of a 0.6 MB page / a 2.3 MB page | 16 / 24 |
| A map of gov.uk (508,000 URLs in 29 files) | 29 |
| 100 pages extracted to JSON with your own model key, ~3,000 tokens each | 300 |
| A 100-page project on the weekly schedule, plain, for a month | ~400 |

## Plans

**Every workspace starts on the Free plan** — 1,000 credits, once, no card. Move to a paid plan from Billing whenever you need more room; what you have built carries over.

| | Free | Starter | Growth | Scale | Enterprise |
|---|---|---|---|---|---|
| Price | $0 | $29 / mo | $99 / mo | $349 / mo | from $1,500 / mo |
| Credits | 1,000, **once** | 100,000 / mo | 500,000 / mo | 2,500,000 / mo | no ceiling |
| Tier | browser | browser | stealth | stealth | stealth |
| Projects | 2 | 10 | 50 | 250 | ∞ |
| Pages per run | 500 | 5,000 | 50,000 | 200,000 | ∞ |
| Crawls at once | 1 (2 interactive) | 2 (4) | 5 (10) | 15 (30) | custom |
| Agent runs at once | 1 | 1 | 3 | 10 | custom |
| Rows in one agent table | 25 | 200 | 500 | 1,000 | custom |
| Sub-accounts | — | — | — | 25 | custom |
| Requests per minute | 60 | 120 | 300 | 600 | custom |
| Past the allowance | stops | buy credits $5 = 15,000 | overage at $0.30 / 1k up to a cap you set; buy credits $5 = 15,000; priority support, DPA | overage at $0.20 / 1k up to a cap you set; buy credits $5 = 25,000; one-month rollover; invoicing | dedicated workers and exits, static IPs, a person to call |
| In INR, + 18% GST | ₹0 | ₹2,499 / mo | ₹8,499 / mo | ₹29,999 / mo | agreed |

Annual: twelve months for the price of ten; credits still arrive monthly. The free tier is a trial, not a hobby plan: the credits do not renew, nothing to buy.

### Beyond the allowance

Three layers, drawn in this order:

1. **The allowance** — what the plan includes, per calendar month. A month a subscription starts or changes plan in gets a blended number: each day at the plan it was on — the old plan's share of the days gone plus the new plan's share of the days left — so a day-28 upgrade is not a full month's credits for three days' money, and a second change the same month builds on the first. What was spent stays spent; the ceiling moves. A subscription that resumes after a halt gets the month's number back: the cycle was paid in full.
2. **Overage** (Growth and Scale) — work continues past the allowance at the plan's per-thousand rate, billed as a line on the next renewal invoice, **up to a cap you set on Billing**: twice the allowance by default, any multiple, or none. Past the cap, runs stop.
3. **Bought credits** (every paid plan) — packs of $5 (₹449 + GST), 15,000 credits on Starter and Growth, 25,000 on Scale. Drawn only after the allowance and the overage cap, never expiring, kept across a plan change and across a cancellation -- the free plan draws on its own allowance, so bought credits wait there until the workspace is on a paid plan again. Bought from Billing; credited the moment the payment is confirmed.

Scale also carries a month's unused allowance into the next month, once.

A limit agreed for your workspace — on a call, in a contract — is set on the workspace itself and always wins over the plan's number, so a change to what a plan includes never takes away something you were promised. **Settings → Limits** shows what applies to you.

## How a run is metered

- **At the enqueue**, a workspace with nothing left is told so — 402 `plan_limit` with a plain reason — before a job exists.
- **At the claim**, the run is given the budget it may spend, so a crawl cannot outrun the balance mid-run. A scrape and a playground fetch get one too, while credits are enforced.
- **Per page**, what is left bounds the climb: an engine dearer than the balance is not tried, and a page it alone could read is recorded `BUDGET_EXHAUSTED` at 0. A page that needs more than is left (a render with 3 credits to go) is not fetched; the run stops there.
- **Per page**, the run's tally; a crawl that reaches its budget stops with `StopReason: credits` and keeps what it read.
- **The ledger** (the Billing screen): one row per one-off (a scrape, a map, a model pass), one row per engine per crawl (upserted as the run goes — with a note when renders were billed at the rung below), and free lines for the things worth showing at zero. Every page row carries the engine that read it, so a bill can be taken apart to the URL.
- **The meter** on the Billing screen is what the ledger adds up to this month; on the free tier, over the workspace's whole life.

## Payments

Payments go through **Razorpay**, and Billing does the rest:

- **Choose a plan** — monthly or yearly, in USD or INR. INR prices are before GST; 18% is added at checkout. USD is the export list, for a customer outside India, with no GST; a workspace whose invoice details name a GSTIN or an Indian state is offered INR only, since a supply to it carries GST whatever the currency. Checkout is Razorpay's: MeshArc never sees your card. The plan is yours the moment Checkout confirms the payment; Razorpay's webhook says the same thing again a moment later.
- **Tax invoices** — every payment gets a numbered tax invoice (`MA/2026-27/0001`, one series per financial year) from Discover Webtech Private Limited, GSTIN 23AAFCD0880K1ZP, Indore: taxable value and, for a customer in Madhya Pradesh, CGST 9% + SGST 9%; for the rest of India, IGST 18%; for a customer paying in USD, zero-rated as an export of services under LUT. Give your legal name, GSTIN (or your state, if unregistered) and address under *Invoice details* on Billing and they appear on every invoice from the next payment. Open it from Payments → *tax invoice*; the browser's Print saves the PDF. Razorpay's own receipt is linked beside it.
- **Credit notes** — a refund against an invoice (the unused days of a mandate replaced on an upgrade, or a refund made by support) gets a numbered credit note (`MA-CN/2026-27/0001`) that names the invoice and reverses the refunded amount's share of its tax. It sits under the payment on Billing → Payments.
- **Change plan** — an upgrade is applied at once and a downgrade when the cycle ends; the month's credits are blended by the day either way. *How* depends on what holds the mandate. Where Razorpay can move the subscription itself — a card issued outside India, a bank mandate (eNACH) — it prorates by the day and charges the difference for the rest of the cycle; a downgrade shows as *Growth until 14 Oct, then Starter* with a *Keep Growth* button. Where it cannot — UPI, or a card issued in India, whose mandate is fixed when it is registered — the change is a new mandate you authorise in Checkout: an upgrade starts now and the unused days of the old mandate are refunded to the account that paid (5–7 working days, with a credit note); a downgrade's new mandate starts when the current one ends and cannot be called off. Changing the currency means cancelling at the cycle end and starting again.
- **Paid by UPI?** A UPI mandate cannot be moved to another plan, so a change is a new mandate to authorise through Checkout: an upgrade starts now and the unused days of the old mandate are refunded pro rata; a downgrade starts when the current mandate ends, and cannot be called off. *Update card* does not apply to a mandate — to pay another way, change the plan and authorise the new subscription with a card.
- **Update card** — Checkout again, on the existing subscription.
- **Cancel at cycle end** — the plan stays until the period ends, then the workspace is on the free plan with its projects and data kept.
- **A failed renewal** — Razorpay retries on three consecutive days; the plan stays meanwhile and Billing says so. If every retry fails the subscription is *halted*: the workspace is on the free plan and runs stop until the payment goes through — *Update card*, approve the request in your UPI app, or the link in Razorpay's email. Your projects, pages and history are kept; nothing is deleted for non-payment. The workspace's owners and admins are emailed by MeshArc on the day the payment fails, on the day it is halted, and a week and a month later; then no more.
- **Renewals above ₹15,000** — the RBI's ceiling for a recurring payment taken without you (Scale monthly, and every yearly plan in INR). A card cannot hold a mandate that size. Pay by **bank mandate** (eNACH, offered in Checkout) and renewals are automatic; pay by **UPI** and your UPI app asks you to approve each renewal — MeshArc emails you three days before. If a renewal is not approved it is a failed renewal, as above.
- **Buy credits** and the **overage cap** — see *Beyond the allowance* above.
- **Payments** — every charge with its Razorpay invoice, on Billing.

For the API: `GET /me/billing` carries the catalogue, the subscription, the bought-credit balance and the receipts; `POST /me/billing/subscribe`, `/confirm`, `/change`, `/keep`, `/cancel`, `/card`, `/credits/buy` and `PATCH /me/billing/cap` are what the buttons call. They need an admin session with a fresh second factor; an API key cannot buy.