Configuration: every setting
A project's config, a one-shot POST /crawl, a POST /scrape (its config and scrapeOptions) and the playground all take the same settings with the same defaults. The Configure screen and the playground rail show them as the same 84 controls, so a request that names only what differs runs what the screen showed. GET /meta returns the full table.
Keys are given by their API name; the app's label follows in parentheses. Default is what you get when the key is absent.
How a setting reaches the engine#
| Where | How to set it |
|---|---|
| A project | PATCH /projects/{id} with {"config": {...}}, or the Configure screen |
| A one-shot crawl | POST /crawl with config: {...}, or the short names limit, maxDepth, includePaths, excludePaths, useSitemap, sitemapMode, crawlMode, allowSubdomains, maxAge, maxTier, delay, concurrency, respectRobots |
| A scrape | POST /scrape with config: {...}, or scrapeOptions: {formats, actions, actionsOn, actionsPaths, onlyMainContent, includeTags, excludeTags, waitFor, waitMs, viewport, maxTier, jsonSchema, parseDocuments, minWords, languages}, plus maxCredits and formats at the top level |
| The playground | The rail; the request sends what differs from the defaults |
On a one-shot POST /crawl or POST /scrape, unknown keys are ignored, not refused, so a newer SDK never fails on an option the API could do without. A project's config (POST and PATCH /projects, and the screens that send it) refuses a key that is not a setting with 422, naming it, so a misspelt key is never taken for one that was set. A value out of range is refused (400, or 422 from a project's config) with the range.
Output formats#
| Key | Meaning | Default |
|---|---|---|
formats | Which bodies a page keeps: markdown (always), text, rawHtml (the page as served, for every page — 5–20× the storage), cleanHtml (after boilerplate, selectors and images, before markdown, without scripts and styles — selectors still resolve), links (every link with anchor text, resolved, typed internal / external / anchor), raw (status, headers, redirect chain and served body), screenshot (a JPEG of the viewport per page, ≤600 kB; forces a render, +1 credit), json (structured fields) | ["markdown", "links"] |
json_schema (JSON schema) | The fields the json format fills from what the page declares — head and Open Graph meta, JSON-LD, microdata, <time>, byline markup: [{name, type, required}], types string / number / date / boolean / url / array. Names match loosely (published_at, datePublished and date are one fact). Unresolved is reported, never guessed | [] |
rows (Read the page as records) | {source, path, selector, columns, key, next, filters}: a page that is a list — a board of roles, a table of recalls, a feed of posts — read as records instead of as one thing. source is json (a dotted path into the body, e.g. data.children[].data), table (the page's markdown tables, headings matched loosely) or list (a CSS selector matching one record). columns are typed as a JSON schema is; key is the column that names a record, or several when no single one does. next: {path, param, max_pages} follows a cursor the body hands out, for a feed nothing links to. filters: {since_days, must_contain, drop} keep records out and are counted. Rows are compared with the previous run's by key: added, changed (one event per column, with both values) and gone — gone withheld when the run reached under 90% of the site: such a row stays listed, marked withheld (one withheld event), until a run sees it again | null (off) |
llm_extract (Fill missing fields with a model) | {connection_id, instructions, only_missing, max_pages, max_chars, max_output_tokens}: a Models connection fills the fields the page did not declare, from the markdown, cached by content hash; 1 credit per 1,000 tokens as the pipeline fee, the model bill is yours. only_missing true asks only for unresolved fields; max_pages caps calls per run (1–2,000), and the workspace's credits cap them too: the stage stops asking once what the crawl left is spent (the run's model tally says stopped: credits), while cached answers still fill their pages. max_chars (0 = as much of the page as the model's context holds) and max_output_tokens (0 = a budget that grows with the schema, an array field being a table) are ceilings for a workspace that would rather spend fewer credits on a long page: past max_chars the blocks naming your schema's fields are sent instead of the page's opening, and a reply that runs out of room is cut at its last complete value rather than lost | null (off) |
heading_style | atx (# prefixes) or setext (underlines) | atx |
link_style | inline ([text](url)), reference (footnotes), strip (text only — smallest). Links are absolute, resolved against the page's address (and its <base href>), as images are; an in-page #anchor and a mailto: stay as written | inline |
Content filters#
| Key | Meaning | Default |
|---|---|---|
only_main_content | Drop nav, header, footer and sidebars — the chrome that repeats on every page — and related-content blocks wherever they sit (related posts, "you may also like", recommendation widgets), so a menu edit or a new post elsewhere is not "the whole site changed". A sidebar inside the page's main or article, a sidebar or related block holding most of the page's words or the main or article itself, and anything include_tags names or preserve_tags keeps, stay | true |
drop_link_menus | With only_main_content, also drop the site furniture a site built from plain blocks with nothing marking it as such. Link menus (w3schools' bars and side menu): a block of at least 8 short links that is mostly links, outside the page's main or article and not most of the page — unless the site names it a side menu (an id or class such as leftmenu, left-nav, sidenav or sidebar, on it or on a block around it that holds no h1 and little more than the menu; a bare menu, nav or toc, or a layout class such as right-sidebar, is not one) and the page has its own h1 and at least 100 words of its own text, not links, beside it (the site's header, footer and copyright lines do not count), as a short reference page's side menu does. An index page's list, with its heading and intro over it, is kept. And a footer that is not a <footer> but is named one by its id or class (footer, site-footer, footerwrapper), with the promos and copyright line inside it; an article's or a card's own footer (article-footer, card-footer, or a footer class repeated in each item of a list) is part of the text and stays. A page that is itself a list of links keeps it, and so does a list of at least three alike records, each with a block of links of its own beside its words (each quote's tags, each product's categories), however many links one item holds. Off for a project, so its comparisons keep reading pages as before (turning it on re-baselines once); a one-off scrape or crawl turns it on (to keep the menus, drop_link_menus: false in a scrape's config, or dropLinkMenus: false in a crawl's scrapeOptions) | false |
strip_boilerplate | Cookie banners, newsletter prompts, "was this helpful", share bars, back-to-top — by vendor markup and by shape | true |
strip_base64 | Inline data-URI images out of the markdown (their alt text stays) | true |
include_tags | CSS selectors to keep exclusively; everything outside them is dropped | [] |
exclude_tags | CSS selectors to drop; exclude wins over include | [] |
preserve_tags | Selectors that survive every kind of pruning — a price table in a sidebar the chrome drop would take | [] |
min_words (Minimum word count) | Off at 0. Set a floor and pages under it are recorded as skipped — stored, out of the change record. A page whose markup has a shape (a listing, a table, a form) is never under the floor, and the home page is never skipped. A short page is marked warnings: ["short"] on its row either way (0–5,000) | 0 |
languages | Two-letter codes; a page outside the list is skipped. Detected from the html tag, headers or text; an undetectable one is never skipped | [] |
Crawl scope#
| Key | Meaning | Default |
|---|---|---|
max_pages (Page budget) | The most pages one run fetches; the plan caps it (free 500 · Starter 5,000 · Growth 50,000 · Scale 200,000). A run that stops at its budget with URLs queued reaches only part of the site, and the diff withholds removals | 100 |
max_depth | Link hops from the seed; 0 is the seed alone (0–10) | 3 |
breadth_per_level | Cap on pages fetched at each depth; 0 is none. A cap silently leaves the rest of a level unread, which the link graph reports as reduced coverage | 0 |
respect_robots | Obey Allow / Disallow (for mesharc, else *) and Crawl-delay; disallowed URLs are recorded as robots, never fetched | true |
allow_subdomains | Follow links to other hosts under the same registrable domain | false |
include_paths / exclude_paths | Globs on the URL path (/docs/*); excludes win; empty include means everything | [] |
sitemap_mode (Sitemap source) | auto (robots.txt, the well-known paths, the seed page's link tags; indexes walked), listed (only sitemaps), off | auto |
sitemaps | The sitemap URLs a listed project reads (up to 20, same site) | [] |
sitemap_include / sitemap_exclude | Globs on a child file's URL or section label (*press*) that choose which files a crawl keeps | [] |
crawl_mode (Crawl from) | sitemap_first (the selected sitemap URLs at depth 1, then links — the only mode that can tell an orphan from a page never reached), sitemap_only (exactly the selected URLs, nothing followed), links (seed and links; the sitemap still informs the link graph and the sitemap diff). A project that turns its sitemap off without naming a mode falls back to links | sitemap_first |
sitemap_incremental | A scheduled run fetches only what the sitemap says moved (added, or lastmod newer than the last real fetch) and carries the rest over from the last full run at no fetch. A full run is forced every sitemap_full_every runs, when fewer than 90 % of entries carry lastmod, or when a carried page is over 30 days old | false |
sitemap_full_every | Full run every N runs of an incremental project (1–365) | 10 |
crawl_delay_ms | The pause between requests to one host; a robots.txt crawl-delay raises it, a rising refusal rate slows it, nothing lowers it (0–60,000) | 1000 |
concurrency | Parallel fetches against the host on the http rungs once the first page has settled which rung works; the delay becomes the gap between rounds. A render stays one page at a time; a robots crawl-delay forces 1. 2–4 is safe on sites that do not rate-limit (1–8) | 1 |
url_list / feeds / patterns | Sources beyond the seed, set on the Sources screen (5,000 URLs; 20 feeds; 20 patterns, 5,000 expansions) | [] |
Images#
| Key | Meaning | Default |
|---|---|---|
image_mode | alt writes the alt text in place of the image; link keeps  with the URL made absolute | alt |
image_exclude | Globs on the image path (*/icons/*, *sprite*) | [] |
image_min_width / image_min_height | Drop images the markup declares smaller (tracking pixels, icons); only attributes are read, nothing is fetched | 0 / 0 |
image_max_kb | Drop inline data-URI images above this | 0 |
alt_fallback | When an image has no alt: filename, caption (the figcaption), nearby (surrounding text), none | filename |
Fetch behaviour#
| Key | Meaning | Default |
|---|---|---|
max_tier (Max tier) | How far a refused page may climb; a page starts where the site is known to answer (the cheapest engine, on a site not read before) and climbs only when refused: auto (as high as the plan allows), http (1–2 credits; a walled page is recorded TIER_LIMIT), browser (a render when needed, 4), stealth (a residential exit 16, a solved widget 22, a rented fetch 40). The plan caps it (Starter: browser; free: stealth on a 200-credit trial, then browser), and on any tier a climb never goes past the credits left | auto for a project made with New project, else browser |
render_js (Render JavaScript) | auto: render only when the http rung comes back an app shell or a short page (under 60 words), to see what its scripts add. A render that only confirms the short page is billed as the http rung; one that finds more text is kept and billed as a render, and the page says so: signals carries render added text and reason how many words the server sent and the scripts added (example.com sends 22 words and a script that adds its notice in five more languages, so it costs 4). always: every page, 4 credits. never: whatever the server sent, at the http rung's price (sets max_tier to http) | auto |
googlebot (Ask as Googlebot) | Add Googlebot's identity to the engines this project may use. A claim about who is asking; sites that verify by reverse DNS are not persuaded; never reached for unless on | false |
user_agent | Where the ladder starts: default (a rotating crawler identity), chrome (a real Chrome with its TLS fingerprint), googlebot (turns the Googlebot setting on) | default |
solve_captcha | Have MeshArc's token solver answer a Turnstile / reCAPTCHA widget on a form the steps submit, before the steps run. Stealth tier; 22 credits when a token is bought. A widget on a page you only read needs no solving | false |
max_credits_per_page (Max credits per page) | The most one page may cost; engines dearer than this are never tried, and a page only they could read is TIER_LIMIT at 0. 0: the tier is the ceiling. maxCredits on /scrape (0–100) | 0 |
max_credits_per_run (Max credits per run) | The most one run may spend; a run that reaches it stops with run cap, keeping what it read. 0: no cap beyond what the workspace has left | 0 |
max_credits_per_month (Max credits per month) | The most the scheduled runs may spend in a calendar month (UTC). A run due once it is spent is skipped (the project says so) and a webhook that asked for budget.reached hears it once that month; Run now is not held to it. 0: no cap | 0 |
wait_for_selector | When rendering, wait up to 20 s for this node before reading; read what is there if it never appears | "" |
wait_ms | A flat delay after load on top of the settle; paid on every rendered page (0–60,000) | 0 |
actions (Browser steps) | Steps after load and before reading: click (with repeat: "until_gone" and max for a Load-more button), clickall, each (over every element a selector matches, with nested steps), type, select, press, wait (ms or selector), scroll (bottom ×N or to a selector). Up to 40 leaf steps; a step that fails is noted on the page and the page is read as it stands; a page that takes steps is always rendered | [] |
actions_on (Run steps on) | start (the seed plus the URL list and pattern URLs — a listing whose Load-more reveals the pages to crawl; a consent click the browser remembers), paths (pages matching actions_paths), all (every page — forces a render for the whole run, 4 credits a page) | start |
actions_paths | Path globs for paths | [] |
fetch_timeout_ms | Abort a fetch after this; recorded as a timeout (1,000–120,000) | 30000 |
fetch_retries | Extra tries on a dropped connection or 500 / 502 / 503 / 504 / 400 / 408 / 409 / 430. A 429 is handled by the pacing; a 403 climbs the ladder (0–5) | 2 |
viewport | desktop or mobile for renders | desktop |
use_proxy (Proxy pool for http fetches) | Route the plain http fetches through MeshArc's datacenter proxy pool (each exit is checked against a direct fetch before it is trusted). No change to the price. The residential exit for renders is max_tier: stealth, a separate thing | false |
geo | auto or a two-letter country: requests from that country through the residential exit; a requirement the ladder honours (stealth) | auto |
cache_max_age_hours | Reuse a page any project of the workspace fetched this recently, at 0 credits, instead of fetching (maxAge on the verbs; 0–720) | 0 |
force_cache | Whether a scheduled run may answer from the cache. Off, a scheduled run always looks | false |
Advanced#
| Key | Meaning | Default |
|---|---|---|
Schedule (schedule on the project, not in config) | manual, hourly, daily (03:00 UTC), weekly (Mondays 03:00 UTC). Change detection needs two runs. A 100-page project is about 400 credits a month weekly, 3,000 daily | The app creates projects weekly; POST /projects and one-shot crawls: manual |
Retention (retention on the project) | How long page content (HTML, text, markdown) and screenshots are kept: 30d, 90d, 1y, forever; the project's two most recent runs, and any older run whose pages a newer run still reuses, keep theirs whatever it says. URLs, extracted fields, hashes and the change record are kept regardless; sitemap snapshots' URL lists follow the window too | 90d (a one-shot crawl is deleted after a day unless kept; kept, 90d unless retention is given) |
webhook_url | Where the project's messages go (http(s)) | "" |
webhook_events | Which of page.changed, run.finished, page.blocked, field.canonical, site.failing, site.recovered, page.orphaned, page.linked, sitemap.added / removed / updated / file_added / file_removed, and the job events | ["page.changed", "run.finished", "field.canonical"] |
webhook_include | ["markdown"]: page.changed carries each added or modified page's markdown beside its URL, up to about 1 MB of markdown a message; a page past that, or not stored, comes with markdown: null, markdownOmitted (too_large or not_stored) and a contentUrl to fetch it from. Subscriptions are sent the message without it. An edit here does not bump the config version | [] |
notify_min_words | A modified page under this many changed words stays out of page.changed; added and removed pages always count | 5 |
chat_url | A Slack, Discord, Microsoft Teams (Workflows), Mattermost or Google Chat incoming-webhook URL (https). The run record posts there when a run finds something — never when it compared clean — in the service's own shape, plus site.failing / site.recovered. Returned masked; a masked value sent back keeps the one stored, "" removes it. An edit here does not bump the config version | "" |
change_digest | {connection_id, focus} — an llm connection of the workspace and a line on what the reader cares about. After each compared run with something in it, the model writes a paragraph on what changed and why it matters, up to five points and a significant flag, from the changed paragraphs, the added pages' openings, the removed paths and the field diffs. Carried by the run email, the chat message, run.finished and the change set. Charged like enrichment, through your own key | null |
login_id | A saved login the runs read the site with, signed in: its id (GET /logins). It must be for this project's site (or a parent of it); it is used on that site and its subdomains only. A run whose login has expired or was deleted stops and says so — it never reads the site signed out instead. "" reads signed out | "" |
parse_documents | Follow links to PDFs and Word (.docx) files and record each one's text as a document — a PDF as one markdown block per page, a Word file as its headings, paragraphs, lists and tables — with a word count and a hash, compared like a page (2 credits; a rendered one 4; up to 400 pages / 25 MB). Older Word (.doc), PowerPoint and Excel files are recognised and recorded as seen, not read (no reader ships yet). A scanned PDF (no text layer at all) is read by OCR, its first 30 pages, and its page says extractMethod: "ocr"; if OCR finds no text either, or fails, the page says so. A file a wall refuses — a challenge or captcha page, or the vendor's own block page, not a site's own 401, 403 or 500 behind it — is asked for once more with the cookies a browser earns on the site, still at 2 credits, when the run has a browser and the project's ceiling and credits allow that rung (a run that earns no cookies tries once); its reason names the wall, not only the status. Off, documents are never fetched — one met by content type is recorded as seen and not charged | false |
Legacy aliases#
use_sitemap (true unless sitemap_mode is off), allow_browser (false when render_js is never) and keep_pages are kept in step automatically for older clients.
Things that follow from a setting#
screenshotinformats, or steps withactions_on: all, forcerender_js: always.render_js: neverforcesmax_tier: http.user_agent: googlebotsetsgooglebot: true.crawl_modeother thanlinksneeds a sitemap: withsitemap_mode: offan explicit mode is refused, an implicit one becomeslinks.sitemap_mode: listedneeds at least one URL insitemaps; a sitemap or a source on another host is refused.- A project's
max_pagesis capped by the plan'smax_pages_per_run;max_tierby the plan's tier, on a scrape, the playground and a map as well.autois the plan's tier.