LangChain

langchain-mesharc puts MeshArc inside LangChain: a document loader that turns web pages into Documents, and two tools an agent can call to search the web and read pages. Its distinctive mode is changes, which loads only the pages that changed since a project's last crawl, so a vector store re-embeds what changed and nothing else.

It is listed in LangChain's document loader integrations, and the source is mesharc-org/langchain-mesharc (MIT).

Install#

pip install langchain-mesharc

Python 3.10 or newer, with langchain-core 1.x. It uses the Python SDK underneath.

Make an API key under Settings → API keys in the app and set it as MESHARC_API_KEY, or pass api_key= to the loader or a tool. Reading, crawling, mapping and searching need a key with write scope; changes works with a read-only key. The free plan includes 1,000 credits, with no card.

The document loader#

from langchain_mesharc import MeshArcLoader

# One page, or several read as one batch
docs = MeshArcLoader("https://example.com/pricing").load()
docs = MeshArcLoader(["https://example.com/", "https://example.com/about"]).load()

# A whole site, or a section of it; pages arrive as they are read
loader = MeshArcLoader("https://docs.example.com", mode="crawl",
                       params={"limit": 100, "includePaths": ["/guides/*"]})
for doc in loader.lazy_load():
    print(doc.metadata["url"], len(doc.page_content))

# The URLs a site declares in its sitemaps, without fetching the pages
urls = MeshArcLoader("https://docs.example.com", mode="map").load()

# What changed on a watched site in its latest run
changed = MeshArcLoader(mode="changes", project_id="<project id>").load()

params go to the SDK call unchanged, so any option the API takes for that call works here, by its API name.

Pages with no content are left out. A page that was blocked or returned a 404 yields no Document unless you pass include_empty=True; its page_status and error_code then say why. In changes mode, removed pages are left out unless you pass include_removed=True, and they arrive with empty text.

Metadata#

Every value is a string, a number or a boolean, so any vector store accepts it.

In changes mode each Document's id is <run id>:<url>.

Keeping a vector store current#

Point a project at the site with a schedule, then after each run load only what changed. Keying each page by its URL makes re-adding a page replace the old copy, and include_removed=True tells you what to delete.

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_mesharc import MeshArcLoader

store = InMemoryVectorStore(embeddings)   # any LangChain embeddings and vector store

def refresh(project_id: str) -> None:
    docs = MeshArcLoader(mode="changes", project_id=project_id, include_removed=True).load()
    gone = [d.metadata["url"] for d in docs if d.metadata["change"] == "removed"]
    fresh = [d for d in docs if d.metadata["change"] != "removed"]
    if gone:
        store.delete(ids=gone)
    if fresh:
        store.add_documents(fresh, ids=[d.metadata["url"] for d in fresh])

Run refresh from the project's run.finished webhook, or on the same schedule as the project. A run that reached less than 90% of the site withholds its removals, so a blocked crawl never deletes pages from your index.

If you split pages into chunks before embedding, give each chunk an id that starts with the page's URL, and delete a page's old chunks before adding its new ones.

Tools for agents#

from langchain_mesharc import MeshArcScrape, MeshArcSearch

tools = [MeshArcSearch(), MeshArcScrape()]

Both return content_and_artifact: the content is the text the model sees, with each page cut at max_chars_per_page (8,000 by default; 0 for no limit), and the artifact is MeshArc's full response. A refused request, such as one with too few credits left, comes back to the agent as an error message rather than stopping the run.

What it costs#

Requests are billed in credits like any other API call: a plain page read is 1 credit, a rendered page 4, and blocked pages and 404s cost nothing. A web search costs what reading its results page costs, and a map costs 1 credit per sitemap file. changes reads pages the project already stored. See Credits, plans and billing.