Connectors and destinations

A connection is a provider plus the credentials to reach it — a bucket and a key, a Postgres DSN, a Mongo URI. It belongs to the workspace, is named, and is tested before it is trusted (POST /connections/{id}/test). Secrets are kept apart from the rest of the config, encrypted at rest, and never returned by the API.

A destination binds a project — or, shared, every project or a chosen set — to a connection and says what to write: the target, which columns, the write mode, the key, whether every page goes or only the ones the change record says moved, and when. A sync is one destination writing one run, recorded with rows written, rows skipped and why, and the location.

Push is free: no credits are charged for writing.

The connectors#

Verified says whether MeshArc has run it end to end against a live instance; the ones marked account are written to the provider's current documentation and are confirmed with a real account on request.

The catalogue also lists a Local disk connector; it writes to MeshArc's own storage and exists for testing, not for your data.

Vector indexes: MeshArc chunks the markdown into paragraph-aligned pieces of about 350 words with a 40-word overlap; your OpenAI-compatible embeddings endpoint (embedding_url + embedding_model, dimensions declared) embeds them. Each chunk is a point with a stable id per (url, chunk), the vector, and the mapped columns as payload with the chunk text and index — never the whole markdown.

A destination#

Up to 10 destinations per project and 20 shared ones per workspace; a sync writes at most 20,000 rows.

The columns#

The default set: url, site, final_url, crawled_at, run_id, project_id, content_hash, markdown, word_count, token_count, title, meta_description, canonical_tag, robots_content, h1_tag, redirect_urls, response_status, block_reason, depth, change_class, added_words, removed_words, changed_fields.

Shaping#

A warehouse has dbt after the load; an application table or a vector index has nothing after the write, so what lands must already be the shape the reader wants. A shape is declarative — no code runs — and lives on the destination, so the same run can land raw in one place and shaped in another. POST /destinations/preview shows the shaped rows of the last run before you save.

{
  "filter": [
    {"field": "status", "op": "eq", "value": "ok"},
    {"field": "path", "op": "glob", "value": "/products/*"},
    {"field": "fields.price", "op": "exists"},
    {"field": "word_count", "op": "gte", "value": 120}
  ],
  "map": [
    {"name": "price_cents", "from": "fields.price", "ops": ["number", "mul:100"], "cast": "int"},
    {"name": "category", "from": "url", "ops": ["path_segment:1"]},
    {"name": "source", "from": "=mesharc"}
  ],
  "drop": ["raw_html"]
}

• filter — conditions every row must pass, all of them (up to 20). Fields are the row's columns, plus status, path and host (visible to the filter even when not written), plus anything under fields. from the page's structured data. Operators: eq, ne, in, not_in, gt, gte, lt, lte, contains, not_contains, matches (regex), glob, exists, missing, empty, not_empty.
• map — columns to add or replace (up to 60), each from a source field (or a literal with =) through a chain of up to 8 functions and a cast (string, int, float, bool, date, json). Functions: trim, lower, upper, number, mul:N, date, host, path, path_segment:N, regex:pattern, split:sep, slice:a:b, replace:from:to, default:value, len, first, join:sep, json, not_empty.
• drop — base columns to leave out after mapping.

What a shape cannot do belongs in the reader's own pipeline; the function set is small on purpose.

One-off jobs#

A destination does not need a project. POST /scrape, POST /crawl and the Playground (Send to) may each name one, and when the job is done its pages are written through that destination's columns, shape and sink — every page, with change_class of baseline, since a one-off has no run before it to compare with. A batch of scrapes is one write, however many hosts it spans. The write is on the destination's own sync history and on the job, under destinationSync.

A one-off crawl writes only where it names. A shared destination scoped to every project (all) does not receive one-off crawls: that scope is about the sites you watch, and a quick crawl turning up in that table would be a surprise. Kept as a project (POST /crawl/{id}/keep), it is written like any other.

Syncs#

GET /destinations/{id}/syncs (and the per-project form) lists every sync: when, which run (and which project's, for a shared destination), rows written, rows skipped and why (filtered out, unchanged, over the row cap), the location written, and an error when the write failed. A failed sync does not fail the run; the Destinations screen shows it and Sync now retries.