Learn
What is RAG?
RAG, or retrieval-augmented generation, is a way of making an AI answer from your own information. Before the language model replies, a search step finds the most relevant passages in your documents, emails or records and adds them to the prompt, so the answer quotes your facts instead of relying on what the model remembers from training.
Updated
How does RAG work?
RAG works in two stages: prepare your documents once, then search them each time someone asks. Anthropic's description of a standard setup:
- Split the documents into chunks, "usually no more than a few hundred tokens" each.
- Embed each chunk: an embedding model turns it into a list of numbers that captures its meaning.
- Store the embeddings in a vector database that can search by meaning.
- Search at question time: the question is embedded the same way, and the closest chunks are found.
- Generate: the best chunks are added to the prompt, and the language model writes an answer from them, ideally citing which chunk said what.
Take "what's our notice period for the cleaning contract?" The search finds the clause in the contract and a Handbook page on supplier terms; the model reads both and answers "30 days, in writing", with links to each.
Why do businesses need RAG?
Businesses need RAG because a language model knows nothing about them. It was trained on public text up to a cut-off date, so it can't know your prices, your refund policy, what a customer said last week or what's in a contract. Without that information it either says it doesn't know or, worse, produces a plausible guess.
RAG fixes this without retraining anything:
| Problem | Without RAG | With RAG |
|---|---|---|
| Your own facts | The model guesses or says it can't know | Answers from your documents |
| Recent changes | Stuck at the model's training cut-off | As fresh as your latest page or email |
| Checking answers | No way to see where an answer came from | Each answer cites its sources |
| Privacy | You paste documents into a chat by hand | Only the relevant passages are sent, for that question |
| Updating | Retraining is slow and costly | Edit the page; the next answer uses it |
What's the difference between RAG, fine-tuning and a long prompt?
They're three ways to get your knowledge into an AI's answer, and RAG is usually the right default for business facts.
| Approach | How it works | Good for | Weak at |
|---|---|---|---|
| Long prompt | Paste everything into the context window | Small, stable collections | Grows costly and slow; stops fitting |
| RAG | Search, then add only the relevant passages | Large, changing collections; answers with sources | Misses when the search does |
| Fine-tuning | Train the model further on your examples | Style, format and narrow tasks | Facts that change; giving sources |
Anthropic's rule of thumb for the first option: if a knowledge base is under 200,000 tokens, "about 500 pages of material", you can include all of it in the prompt with no need for RAG. A small business's Handbook may fit; its email archive won't.
What makes RAG work well or badly?
RAG is only as good as its search and its sources. The common failures, and what helps:
- The search misses. Meaning-based search can overlook exact terms such as an order number or an error code. Anthropic's example is a support query for "Error code TS-999" that embeddings miss and keyword search finds; it combines both, and reports that its "Contextual Retrieval" method cut failed retrievals by 49%, and by 67% with reranking.
- Chunks lose their context. A passage reading "the fee rises by 5%" is useless without knowing which contract it's from. Adding a line of context to each chunk helps.
- Sources are stale or contradictory. Two versions of the refund policy give two answers. Keep one current page per topic, with an owner and a review date.
- Permissions leak. A search over everything can surface a document the person asking shouldn't see. The search must run as that person.
- Retrieved text gives orders. OWASP lists retrieved content, such as a web page or an email, as a route for indirect prompt injection. Treat retrieved text as data, never instructions.
When does a small business need RAG?
A small business needs RAG as soon as it wants an AI to answer from more than a few documents, which in practice means almost any AI that knows the business. You rarely build it yourself: help desks that answer from a knowledge base, AI search inside a workspace and assistants that search your email all use retrieval underneath. The questions to ask a vendor are what it searches, whether answers cite sources, whether the search respects each person's permissions, and how quickly an edited page changes the answers.
How does Mojobase use RAG?
Mojobase's AI retrieves before it writes. One search covers mail, calendar events, files and documents, tasks and Handbook pages, ranked together and run as the person asking, so it only ever finds what they could open. Quick actions (⌘K) answers questions such as "what notice does the cleaning contract need?" with quotes and links to where each fact came from.
The Handbook adds labels to every answer it drives: Approved, Draft or Changed by AI since approval. When no page covers the question, it says so rather than guessing. The support AI in Customers answers only from your published Handbook pages and hands anything else to a person. More on the Handbook page.
Further reading
Questions people ask
What does RAG stand for in AI?
Retrieval-augmented generation. The term comes from a 2020 research paper by Patrick Lewis and colleagues, which combined a language model with a search over a large collection of documents so answers could draw on specific, up-to-date text.
Is RAG the same as fine-tuning?
No. Fine-tuning changes the model by training it further on your examples, which is slow to update and doesn't give sources. RAG leaves the model alone and hands it the right documents at question time, so updating a page updates the answers at once.
Does RAG stop AI hallucinations?
It reduces them, it doesn't end them. If the search finds the right passage, the model usually answers from it. If the search misses, or the documents are wrong or out of date, the model can still answer confidently. Asking for sources makes misses easy to spot.
Do I need RAG if the AI has a big context window?
Sometimes not. Anthropic suggests that if your knowledge base is smaller than 200,000 tokens, about 500 pages, you can put all of it in the prompt. Beyond that, or when it includes years of email, you need a search step.
What is a vector database?
A database that stores text as embeddings, lists of numbers that capture meaning, so it can find passages similar in meaning to a question even when they share no words. It's the most common way RAG systems search.