How to keep a RAG knowledge base fresh without re-embedding everything

Most RAG pipelines are built once and refreshed by brute force: re-crawl the site every night, re-chunk everything, re-embed everything, overwrite the index. It works, in the sense that the answers are eventually current. It is also wasteful, and it hides three bugs that a nightly rebuild does not fix.

This post covers what actually goes stale, what a full rebuild really costs (less than you might think in embeddings, more elsewhere), a do-it-yourself content-hash approach you can run with any scraper, and how we do it in MeshArc. You can use the first half without ever signing up for anything.

What goes wrong when a knowledge base ages

A stale chunk does not announce itself. It still sits close to the query in vector space, so retrieval still ranks it highly and the model still quotes it with confidence. Three failure modes are worth separating:

1. Stale chunks. The page changed; the index holds the old wording. The assistant recommends a deprecated flag or an old price.
2. Zombie chunks. The page was deleted, but its vectors were not. Nightly upserts never remove anything, so deleted pages live on indefinitely unless something deletes them on purpose.
3. Flicker re-embeds. A cookie banner, a "last updated 3 minutes ago" line or a rotating testimonial makes every page look modified on every run, so a "changed pages only" pipeline quietly turns back into a full rebuild.

There is a fourth, nastier one: the blocked crawl that deletes your index. If the site rate-limits you halfway through a run and your pipeline treats "not seen this run" as "removed", you wipe half the knowledge base. Any deletion logic needs a guard against partial runs.

What a full nightly rebuild costs

Here is the arithmetic for a mid-sized documentation site. The assumptions are ours and stated so you can swap in your own:

• 2,000 pages, averaging 1,200 words each.
• Chunks of about 350 words with a 40-word overlap, which gives roughly 4 chunks per page and 8,000 chunks in all.
• About 1.3 tokens per English word.
• OpenAI text-embedding-3-small at $0.02 per million input tokens (OpenAI's model page, checked on 23 October 2026).

That is roughly 2.8 million embedded words, or 3.7 million tokens, per full rebuild: about $0.07. Thirty nights is about $2.20 a month.

So, to be fair to the brute-force approach: with a small embedding model, the embedding bill is not what hurts. A larger model, a bigger corpus or a hosted embedding service with per-request pricing changes that, but the real costs of rebuilding everything are usually elsewhere:

• Fetching. 2,000 pages × 30 nights is 60,000 page reads a month, every one of which can be blocked, rate-limited or billed.
• Index writes. 8,000 chunk upserts a night, most of them writing identical vectors over identical vectors. Managed vector databases tend to price writes, and self-hosted ones still pay for them in compaction and I/O.
• Correctness. A rebuild that overwrites by ID still leaves zombies behind, and a rebuild that drops and recreates the index has a window in which the assistant answers from nothing.

The fix for all of these is the same: find out what changed, and touch only that.

The do-it-yourself version: hash, compare, delete deliberately

You do not need any particular vendor for this. The pattern is:

1. Normalise each page's text (strip boilerplate, collapse whitespace) and hash it.
2. Compare against the hash you stored last time.
3. Re-chunk and re-embed only pages whose hash moved; give every chunk an ID that starts with the page URL.
4. Before adding a changed page's new chunks, delete all of its old ones. The chunk count can shrink, so overwriting by ID is not enough.
5. Delete pages that have disappeared, but only if the run reached most of the site.

import hashlib, json, re
from pathlib import Path

STATE = Path("hashes.json")

def normalise(text: str) -> str:
    text = re.sub(r"Last updated[^\n]*", "", text)   # add your own volatile lines
    return re.sub(r"\s+", " ", text).strip()

def plan_refresh(pages: dict[str, str], expected_urls: int):
    """pages: {url: markdown} from this run. Returns what to embed and delete."""
    old = json.loads(STATE.read_text()) if STATE.exists() else {}
    new = {u: hashlib.sha256(normalise(t).encode()).hexdigest() for u, t in pages.items()}

    changed = [u for u, h in new.items() if old.get(u) != h]
    removed = [u for u in old if u not in new]

    # Guard: a run that saw under 90% of the site must not delete anything.
    if len(pages) < 0.9 * expected_urls:
        removed = []
        new = {**old, **new}          # keep unseen pages' hashes for next time

    STATE.write_text(json.dumps(new))
    return changed, removed

# changed -> delete chunks with IDs starting f"{url}#", then embed and add new chunks
# removed -> delete chunks with IDs starting f"{url}#"

Two honest caveats. First, this still fetches every page every night to compute the hashes; it saves embedding and index writes, not crawling. Second, the normalise function is where the work lives. Every site has its own flicker, and you will be adding regexes for months.

If you are already on LangChain, its indexing API (a record manager plus index() with incremental cleanup) does the hash-and-skip part for documents you hand it. It does not fetch pages or know whether a crawl was partial, so the first and last steps above remain yours.

Skipping the fetch as well: sitemap lastmod

Many sites publish a lastmod date per URL in their sitemap. If it is trustworthy, you can skip fetching pages that have not changed at all. "If" is doing real work there: plenty of sitemaps stamp every URL with today's date, and some never update it. A sensible incremental crawler checks how many entries carry lastmod, falls back to a full crawl when coverage is poor, and runs a full pass periodically anyway to catch pages whose dates lie.

How MeshArc does it

This is the part where we describe our own product, so weigh it accordingly. Everything below is from our docs and pricing page.

A project is a site plus its settings, run on a schedule (hourly, daily, weekly or manual). Every run is diffed against the previous one, page by page: added, modified, removed, same, withheld. Boilerplate such as cookie banners, newsletter prompts and share bars is stripped by default, and a modified page with fewer than notify_min_words changed words (default 5) does not count as changed.

With sitemap_incremental on, a scheduled run fetches only pages the sitemap shows as new or with a newer lastmod, and carries the rest forward from the last full run. A full run is forced every sitemap_full_every runs (default 10), whenever fewer than 90% of entries carry lastmod, or when a carried page is over 30 days old. Carried-forward pages cost 0 credits. So do refused pages, 404s and 410s.

Removals are only reported when a run reached at least 90% of the site. Below that, missing pages are withheld, not removed, so a blocked crawl cannot empty your index.

pip install mesharc langchain-mesharc

from mesharc import MeshArc

arc = MeshArc()  # reads MESHARC_API_KEY

project = arc.projects.create(
    "https://docs.example.com",
    name="Docs KB",
    schedule="daily",
    config={"include_paths": ["/docs/*"], "sitemap_incremental": True},
)

run = arc.runs.start(project["id"], wait=True)   # first run is the baseline
record = arc.changes(project["id"])
print(record["change"]["counts"])

Then, after each scheduled run (from the run.finished webhook or a cron of your own), load only what changed and apply it to any LangChain vector store:

from langchain_mesharc import MeshArcLoader

def refresh(store, project_id: str) -> None:
    docs = MeshArcLoader(mode="changes", project_id=project_id,
                         include_removed=True).load()
    gone = [d.metadata["url"] for d in docs if d.metadata["change"] == "removed"]
    fresh = [d for d in docs if d.metadata["change"] != "removed"]
    if gone:
        store.delete(ids=gone)
    if fresh:
        store.add_documents(fresh, ids=[d.metadata["url"] for d in fresh])

That keys one document per URL. If you split pages into chunks before embedding, use chunk IDs that start with the URL and delete a page's old chunks first, exactly as in the DIY version. The LangChain integration docs have the details.

If you would rather not run that code at all, MeshArc can write straight into Pinecone, Qdrant, Weaviate or pgvector as a destination. It splits Markdown into paragraph-aligned chunks of about 350 words with a 40-word overlap, embeds them with your own OpenAI-compatible endpoint and key (we add no markup on embeddings), and by default writes only pages classed as added, modified or removed. Destinations are included on every plan.

The same docs site, priced

Back to the 2,000-page site, read daily on our Growth plan ($99 for 500,000 credits, so about $0.20 per 1,000 plain fetches). Assume the pages fetch plainly at 1 credit each, and that around 2% change on a typical day — your site's real churn will differ, which is the point of measuring it.

Approach	Page reads a month	Credits	Cost on Growth
Full re-crawl every night	60,000	60,000	~$11.88
sitemap_incremental, ~40 changed pages a night, plus 3 forced full runs	~1,080 + 6,000	~7,080 (plus sitemap reads at 1 credit per file)	~$1.40

Embeddings follow the same shape: only the ~40 changed pages a day are re-embedded, instead of all 2,000. Pages that need a real browser cost 4 credits rather than 1, so multiply accordingly; our Firecrawl pricing breakdown works through the per-rung maths in more detail.

Where this does not fit

• One-off corpora. If you index a site once for a demo, none of this matters. Any scraper will do.
• Sites without a usable sitemap. Incremental fetching needs lastmod. Without it, MeshArc still diffs and still writes only changed pages to your store, but it has to fetch every page to know.
• Logged-in content. Pages behind your own SSO are better pulled from the source system (Confluence, Notion, a CMS export) than scraped.
• Self-hosting. MeshArc is not open source. If you need the crawler on your own hardware, the DIY approach above with an open-source crawler is the honest answer; our comparison page lists the trade-offs by vendor.

Try it on your own docs

The quickest test is to point a project at a site you already index, let it run twice, and look at the change counts: that number, not the page count, is what your pipeline should be embedding each night. The RAG pipeline use case has the overview; if your assistant reads whole sites at once, the llms.txt generator is a free alternative to embeddings altogether.

Try a page in the playground · Start free — 1,000 credits, no card