Using the app
The app is at https://mesharc.dev. Everything it shows comes from the API; nothing is computed in the browser that the API does not also return.
Accounts and workspaces#
- Sign up (
/signup) with a work email and a password. The first account of a workspace is its owner. A six-digit verification code arrives by mail (30-minute lifetime); until it is entered you can look around, but creating a project, starting a run, issuing a key or inviting someone answers403 email_unverified. The shell shows a banner with a link to enter the code. - Sign in (
/login). Sessions last 30 days and end after 14 idle days; "Keep me signed in" is the default.Profile → Sessionslists them and ends any. An account with two-factor authentication on is asked for a code after the password. - Forgot password (
/forgot) mails a reset link;Profilechanges it while signed in. - Google or GitHub. Where the deployment offers them, Continue with Google and Continue with GitHub sit above both forms. The first time, a MeshArc account is made with the provider's verified email address — verified already, so no code is mailed — and its own workspace, or the invited one. If an account with that address already exists it is linked to it, and a mail says so; for a Google account that is not a Gmail address or managed by a company (Google Workspace), link it from Profile instead. An account with two-factor on is still asked for its code.
Profile → Sign-in methodslists the linked accounts, links another and removes one — never the last way in. An account made this way has no password until one is set through Forgot password. - Workspaces. An account can belong to several; the sidebar switches.
Settings → Workspacerenames it. - Members and roles (
Settings → Members):owner>admin>member>viewer. A viewer reads everything. A member creates and configures projects, starts runs, scrapes, crawls and maps, reviews changes, tests webhooks and syncs destinations. An admin also issues and revokes API keys, sets the rate ceiling, deletes projects, rotates webhook secrets, manages connections and destinations, invites and removes members, sets roles, renames the workspace and opens the billing portal. The owner is the admin who cannot be removed. Invites go by mail with a link that lives seven days; an invite names the role. - Profile: name, time zone, two-factor authentication, whether you receive run emails (a run record when a project finishes) and the weekly digest (Monday from 08:00 in your time zone: every project's week in one message).
- Notifications on screen. On a server with no mail host set up (a self-hosted or development install), run records and monitor alerts are not emailed: they appear as popups in the bottom corner of the app instead, newest first, up to a week old. Open goes to the run's changes or the monitor; × dismisses one. Where a form offers email, it says when this is how it will arrive.
Two-factor authentication#
A second factor is a six-digit code from an authenticator app — Google Authenticator, 1Password, Authy, Microsoft Authenticator, anything that does time-based codes — asked for after the password every time you sign in.
- Turning it on (
Profile → Two-factor authentication → Turn on): enter your password, scan the QR code with the app (or type the key it shows), then enter the code the app displays. Every other session on the account is signed out, since none of them showed a factor. You are then handed ten recovery codes; keep them where you keep passwords, because they are shown only once. - Recovery codes sign you in when the phone is lost. Each works once; the sign-in screen has a "Lost the phone?" link that takes one instead of an app code. After a recovery sign-in the Profile shows how many remain, and
New recovery codesmakes a fresh set of ten (against a current app code) — the old set stops working. - Confirming it's you. With two-factor on, a few actions ask for the code again if the session last showed it more than ten minutes ago: changing the password, turning two-factor off, making new recovery codes, creating an API key, changing a member's role or removing them, and changing the workspace's two-factor rule. A small dialog asks for the code and the action then goes through.
- Turning it off takes your password and a current code. A mail goes to the account's address whenever two-factor is turned on or off, when a recovery code is used, and when new codes are made.
- Without a password (an account that signs in with Google or GitHub), turning two-factor on or off asks for a sign-in from the last ten minutes in place of the password; sign in again if it has been longer.
- Resetting a forgotten password on an account with two-factor also asks for an app code or a recovery code: an inbox alone is not enough to take over the account.
- Requiring it for a workspace (
Settings → Workspace → Require two-factor authentication): an owner — who has it on for their own account — can make it a condition of acting. A member without it can still sign in and look around, but creating a project, starting a run, issuing a key or changing anything answers403 mfa_requireduntil it is set up; the shell shows a banner with the link, and the members list marks who has it on. - API keys are unaffected by any of this: a key was made by someone who had already passed the gate, and a program cannot read a phone.
API keys#
A key is shown once when it is created. It carries scopes (read < write < admin, matching the member roles), optionally a set of projects it may see (any other project — or another workspace's — answers 404, never 403), an expiry, and its own rate limit (rpm), reported on every response as X-RateLimit-Limit / -Remaining / -Reset. Keys start with mesharc_. Revoking is immediate.
The workspace's own ceiling is max_rpm (Settings → Rate ceiling, PATCH /me); concurrency scales with it so the rate to a host never exceeds it.
The playground#
One URL through the ladder, with every setting on the left as a request field. Four modes:
| Mode | What happens | API |
|---|---|---|
| Single page | One page, synchronously; the tabs (Markdown, Text, Clean HTML, Fields, Screenshot, Links, Images) follow the formats you chose, and Request says the rung that won, the engine, the status, the words and the content hash. A page that could not be read says why, with the response's evidence. What it cost is on Billing's ledger | POST /scrape |
| Crawl site | A one-shot crawl of the site with the page budget and depth from the rail; poll its pages; Keep promotes it to a project | POST /crawl, POST /crawl/{id}/keep |
| Map URLs | The site's URLs from its sitemap tree | POST /map |
| Search | Web or news results for a query: Google's organic results, DuckDuckGo's and then Bing's when Google refuses or reads short, with country, language, freshness, the sites to keep to or leave out, and the results page (1–10). Scrape each result to markdown reads every result's page too. The header says how many results, what it cost and which engine answered; Save and schedule keeps it as a monitor; All searches lists the past ones | POST /search, GET /search |
The rail's settings are the same 84 controls as a project's Configure screen — see Configuration for each one; the request sends only what differs from the defaults, and the code snippet (curl, Python, Node) is a copy of the request that just ran.
Agents#
Agents is a chat. On the left, the workspace's conversations, the latest first, grouped by day (Today, Yesterday, Earlier this week, Older), each with its messages, credits and when — or working while a message runs — and a filter; under them the Tables run row by row. + New opens a new conversation: write what you want found, or pick an example. Start from, under the message box, takes the sites it should read first (optional; without one it searches the web), and a site your message names is offered as Start from … in one click. The agent is only here: the Playground's old agent links open a new conversation with their example filled in.
In a conversation, each message is one agent run with its own status, budget, credits, steps and Cancel — billing is per message, as for any run. While a message works, its steps appear as it takes them (searches, pages read, maps, each with its cost), and the line under them says what it is doing now — "Working out what to do next…", "Reading example.com/pricing…"; the model's share closes the list ("2 model steps · 2,060 tokens", and its credits). When it answers, they fold into one line ("5 steps · searched twice · read 4 pages"), opened with a click. The answer shows its sources numbered, a link in the text cites its source's number, and Copy, Download, Full trace, Keep and re-run (not on a follow-up or a continuation) and — at a credit stop — Continue with a new budget sit under it. Proposals the message made appear under it with Approve and Dismiss.
Ask a follow-up in the box at the bottom: it remembers everything asked and found in the conversation, and pages already read in it are read again free (the message says how many it could reuse). The chips under the box — Start from, Answer, Budget, Steps, Sites, and Model, Proposals, Logins and Data when set — are the next message's options, kept from the last message until you change them; one changed from the default is marked, and a chip opens them all. A conversation of three messages or more opens with all but the latest folded under one line naming them ("↑ 2 earlier messages: …"); a click shows them, and what you ask after stays in view. You can send once the last answer has finished, or cancel it. API request shows the next message as POST /agent/{id}/follow-up. A conversation holds up to 30 messages and is kept 7 days after its last one.
Run on a table asks one question of every row of a table (POST /agent/batch):
- The table — upload a
.csvor paste one; its first line names the columns, and the first rows are shown. - What each row asks — the prompt, with a column put in as
{{column}}(click a column to insert it). Start each row at picks the column holding each row's URL; with Keep each row on its own site ticked, each row reads only its own site. Without it, a warning says the rows may read any site. - The answer's columns — the fields each row's answer adds, or free text.
- What it may spend — the most credits a row, and how many rows run at once. Under Advanced, the model, the steps a row and the data providers every row may look things up with.
Check the table asks the server what would run: how many rows, which cannot run and why, the first prompts as their rows will ask them, and about what it will cost against your balance. Run starts the rows that can. The batch's page fills in as rows end — each row's values, its status and its answer as new columns, with the page each value was read on and a link to the row's own run — and downloads as CSV or JSON lines. Cancel stops the rows not yet finished.
Projects#
The workspace table: every project with its pages, reach, blocked share, pending changes, schedule and last run. Select several and Run. New project takes a seed (a domain or a URL), an optional name and a page budget; a new project runs weekly and the form says what that costs ("about 400 credits a month at 1 a page"). New from a list creates many from seeds with a template project's config.
Each project has these screens.
Overview#
Stat cards (pages tracked, pending review, last-run coverage, credits), "what changed in the last run" with field-change chips and the top changed pages, a health panel (healthy / failing / recovered from the last two runs), and recent runs. Run now starts a run.
Sources#
Where the project's URLs come from:
- The seed and its links.
- The sitemap tree — discovered from robots.txt, the well-known paths and the seed page's link tags, indexes walked to their children; or only the files you list (
sitemap_mode: listed); or off. Every file with its section label, URL count andlastmodrange; a diff against the last snapshot (files and URLs added, removed, updated). Pick the sections a crawl keeps (sitemap_include/sitemap_excludeglobs on the file URL or section) and see how many URLs the selection keeps before a run (Preview). - URL list — up to 5,000 exact URLs, fetched as given, never filtered by the path globs.
- Feeds — up to 20 RSS/Atom feeds, read each run; every entry is queued first.
- Patterns — templates expanded into URLs:
/products/{1..500},/docs/{intro,setup,faq}; capped at 5,000 expansions.
The scope preview shows pages per depth with the excluded and beyond-budget bands, and an estimate (~340 pages · ~1.4M tokens · ~4 min).
Configure#
The whole configuration as 84 controls in seven groups — Output formats, Content filters, Crawl scope, Rows, Images & media, Fetch behaviour, Advanced — each with a tip that says what the engine does and what it charges; a JSON pane that round-trips to the controls with line-level parse errors; Dry run (resolve sources and sample URLs without spending); a staged-change diff; Apply to… (copy the saved config onto other projects by group); Webhook deliveries (every message sent, attempt by attempt, with resend); the signing secret and a Send test; a chat channel (a Slack, Discord, Teams, Mattermost or Google Chat incoming-webhook URL — the run record posts there when a run finds something, never when it compared clean; the URL is shown masked once saved); the change digest (a model and a focus line); and the danger zone (delete the project). Configuration has every control with its meaning and its default.
Pages#
Every page of a run with its verdict, engine, tier, credits, words, status and how it was found (seed / sitemap / link / URL list / feed / pattern). Filter with the query language (status:403 depth:>2 flag:orphan). Two search modes scan the run on the server: content (words and "phrases" over the markdown) and selector (a CSS selector or XPath over the stored HTML). A selection can be re-crawled now (a scoped run), exported (JSONL / CSV) or excluded (added to exclude_paths). Removed lists pages that were in an earlier run and not this one.
Page detail#
Five tabs: the rendered markdown, the sandboxed HTML, head fields (title, description, canonical, robots, h1, Open Graph, Twitter card, JSON-LD), structured fields with what resolved each one, and versions across runs with the dual-hash diagnostic (content hash vs HTML hash: a page whose HTML changed but whose content did not is a template edit, not a content change). A screenshot when the project takes one.
Changes#
The change record of a run against the one before:
- Field changes — title, description, canonical, robots, h1, Open Graph, Twitter card, lang — with the before and after, ranked.
- Content diff per page — word-level, side by side or unified, a volatile-block filter (dates, counters, "3 comments"), a section rail.
- Markup comparison — a noise filter, the DOM tree, selector-breakage repointing ("
.pricematched 40 pages last run and 0 now; the same nodes now match.product-price"), ignore rules. - The coverage banner: a run that reached under 90 % of the site withholds removals, and the withheld URLs are listed under it.
- Reviewed marks per change.
- The digest, when the project has one (Configure → Advanced → Change digest, a model from Connectors → Models and a line on what you care about): a paragraph on what changed and why it matters, up to five points, and whether any of it is significant — written by the model from the changed paragraphs, the added pages' openings, the removed paths and the field diffs, on your own key. The same paragraph opens the run email and the chat message.
Rows#
For a project that reads its pages as records (Configure → Rows): every record its pages hold, what moved on the last run, and the records a run stopped finding. A record opens its own history — when it was first and last seen, and every column that ever moved, with the run that moved it. A board of roles, a table of recalls, a schedule of charges and a feed of posts are all lists of records, and watching them as pages only ever says "something changed". Exported as the rows and row-events datasets, and carried by the row.changed webhook.
Runs#
History with status, trigger (manual / scheduled / API / recrawl), pages, credits, stop reason and duration; the live log with time and level while a run is on (polled every 2 s); the engine breakdown; grouped errors; Cancel.
Destinations#
The project's destinations and the shared ones that include it, each with its target, mode, columns, shape and sync history ("synced 4 h ago · 84 written, 1,190 skipped"). Sync now writes the last run by hand. How a destination is set up, and how rows are shaped, is under Connectors and destinations.
Connectors#
Workspace-level connections: a provider plus the credentials to reach it, named, and tested before it is trusted. The catalogue (21 providers) with a connect wizard per provider, then per-connection overview, mapping and settings. Secrets are never returned by the API and are encrypted at rest. The Models kind (any OpenAI-compatible or Anthropic chat endpoint on your key) is what a project's LLM enrichment stage uses. The Data providers kind — People Data Labs (company data) and Financial Modeling Prep (financial data), each on your own key — is what the agent looks things up with: tick one under Advanced → Data providers in the agent's options or a table's. A lookup costs no MeshArc credits (the provider bills your key), appears in the message's steps, and its record is one of the answer's sources. Neither a model nor a data provider can be a destination.
Shared destinations#
One target for many projects: one target that every project (scope all) or a chosen set (scope projects) writes after each of their runs, with site, project_id and project_name as columns so one table holds many sites. A shape (filter / map / drop) previewed against the last run before saving.
Exports#
Datasets streamed on request as JSONL or CSV: pages, markdown, changes, fields, sitemap — per run or per project, or restricted to a selection from the Pages screen. Two text files for a model: llms.txt (the usable pages as links with descriptions, by section — what an assistant reads to know what the site holds) and llms-full.txt (every usable page's markdown in one file, under its title and source URL — the corpus). Also every project's webhook and chat channel with its health from recent deliveries (healthy / failing / paused / idle), and every destination with its last sync.
Monitor#
Proposed by agents, at the top when there are any: what agent runs proposed to watch — a site to crawl, a search or a question to keep on a schedule — each with how often, why the agent proposes it, about what it would cost a month and the run that proposed it. Approve makes it — a monitor's first run starts at once, a project crawls at its first scheduled time or when you run it; Dismiss deletes it. Nothing runs or is charged before that. An agent run proposes only when May propose things to watch is ticked in the message's options; its own page lists what it proposed, decided or not, with a link to what was made.
Scheduled, under it: everything that runs on a schedule — crawl projects with a schedule, kept searches and kept agent runs — one row each, filtered by kind: what its last run found, when the next one is (or paused), what it has spent this month against its cap and about what a month costs, and who is told. In place: Run now, Pause / Resume (the schedule is kept; resumed, a slot that passed runs once), Edit (a project's Configure, a monitor's page), History, and Delete, which asks for the name to be typed.
Under it, Triggers: every trigger in the workspace, what it watches and does, and its last fire. A project's triggers are added on its Configure screen, a monitor's on its page: when a run changes something that matches — pages of a kind, under a path, a field, enough words, or what the digest calls significant; a search's new results; an agent's records — it starts an agent from a prompt you write (with the change filled in), re-crawls the changed pages, runs a monitor, or re-checks only the records of an agent monitor that were read on a changed page. Each has a wait between fires, a most a day and a monthly budget; History lists every fire and what it started, or why it was skipped, and Preview shows what the latest run would do.
Below them, what is queued and running for the workspace, each lane's queue depth and oldest wait, sparklines of pages per day.
In the dialog that keeps a search or an agent run, Most credits a month starts at about what a month costs, rounded up (3,412 becomes 3,500), so a month that runs past what was expected stops there; change it, or clear it for no cap. A search or agent run kept with Save and schedule opens at its own monitor page: its schedule, caps, who is told, and every run with what changed since the last one that answered. When an agent's answer is a list of records, the dialog offers Follow each record, named by a column you pick (the monitor page's Records line changes it later, and can write the records to a destination after each run). The page then shows the Records — every one, what moved, and the ones no longer there, each value with the page it was read on. A record one answer missed is shown as withheld rather than gone: it is gone only when two answers in a row lack it.
Billing#
The plan and what it includes, this month's meter (credits used of the allowance; the free tier's is a lifetime allowance), the ledger (every charge: a scrape, a map, a crawl's rows per engine with a note when renders were billed at the rung below, model passes, free lines for cached pages), the price list, and the billing portal (invoices, the card on file, plan changes). Credits, plans and billing explains what a page costs and what each plan includes.
Saved logins#
Some sites show their content only to an account: a supplier's portal, a members' directory, a paid database, your own tools. A saved login lets a project, a scrape or an agent run read such a site signed in. It is switched on per deployment (MeshArc's own: ask us), and only owners and admins can save, see or delete one.
Saving one. Settings → Logins → Add login: name the site (portal.supplier.com), tick the terms, and press Open the browser and sign in. A real Chrome opens on MeshArc's servers and its screen is shown in the app: click, type, scroll, paste, use the address box or Back as in any browser, and sign in as you normally would — including a code from your phone. When you are in, press Save login. MeshArc keeps the signed-in session (the site's cookies and local storage), encrypted, and closes the browser; your password went into the site's own form and is never kept. The browser closes by itself after a few minutes, or sooner with nothing clicked or typed, and each person has one open at a time.
Using one. Pick it as Saved login under Configure → Fetch behaviour (a project) or in the playground's rail (a page). The agent needs nothing picked: it reads public pages signed out, as anyone sees them, and signs in where a page needs your account, with the saved login for that page's site. One message can read a site's public pages signed out and its account pages signed in, and use several logins; the page's URL decides which. Saved logins under Advanced in the agent's controls turns signing in off, or keeps it to one login, and each page read signed in says so in the message's steps ("signed in as …"). To have a page read signed in, say so in your message ("use my login for …"). A login is used on its own site and that site's subdomains, and never sent anywhere else. A page rendered with a login gets a browser of its own, never one another run shares.
When it stops working. Sites end sessions. When a run meets the site's login page again, the login is marked expired, the workspace's owners and admins get an email, and runs that use it stop — saying so — until it is saved again: Sign in again opens the live browser on its site, and what you save there renews that login, so the projects, monitors and runs that use it carry on. A run never quietly reads the site signed out instead. An agent answer, or an agent monitor's run, that needed a saved login it could not use says so under it — which login, and that its site was read signed out — whatever the answer itself says; on Automatic, a login saved again while the run goes on is used from then on. Test fetches the site's front page with the login and says whether the login page came back; Expire stops using one now; Delete removes it, and the projects that named it stop until another is chosen.
Settings#
Members and invites (with whether each has two-factor on, and the workspace's two-factor rule), API keys, saved logins, the rate ceiling, operator limits (what the plan and the operator allow: projects, pages per run, concurrent jobs, tier, rpm), the ladder (/meta), the proxy pool with per-exit parity and success, and usage by day and by engine with the blocked share.