What Is RAG, and When Does a Company Knowledge Base Need It?

RAG (retrieval-augmented generation) is a way of making an AI model answer from your own documents: the system first searches your content for the most relevant passages, then gives those passages to the model and asks it to answer using only them, with sources. A company knowledge base needs RAG when its content is too large to fit in a single prompt, changes often, must respect who can see what, and has to produce answers people can check. If your content is small, stable or mostly structured data, simpler options often work better.

What is RAG?

Retrieval-augmented generation is a pattern in which a language model’s answer is grounded in passages retrieved from a defined set of documents at the moment a question is asked.

A model on its own answers from what it learned during training. It knows nothing about your returns policy, your supplier terms or last month’s update to the shift rota procedure. RAG closes that gap without retraining anything. The model stays general; the knowledge stays in your documents, where you can update, remove or restrict it.

The practical result is an assistant that can say “the warranty period for spare parts is 12 months, see section 4.2 of the after-sales policy” and link to the passage, rather than a confident guess.

How does a RAG system work, step by step?

A typical RAG pipeline has two halves: preparing the content, and answering questions.

Preparing the content:

1. Collect the sources. Policies, procedures, contracts, product sheets, past tickets, intranet pages, shared drive folders.
2. Clean and split. Convert each document to text and split it into passages, usually by section or heading, keeping the title, date and owner with each one.
3. Index. Turn each passage into an embedding, a numerical representation of its meaning, and store it in a vector database or search index alongside its metadata and access rights.
4. Keep it in sync. When a document changes or is deleted, update or remove its passages automatically.

Answering a question:

1. Search. Convert the question into the same representation and find the closest passages, usually combined with keyword search so exact terms such as part numbers are not missed.
2. Filter by permission. Drop anything the person asking is not allowed to see.
3. Rank. Reorder the remaining passages so the most relevant few go to the model.
4. Generate. The model writes an answer using only those passages and cites them.
5. Decline when unsure. If nothing relevant is found, the assistant says so instead of filling the gap.

When does a company knowledge base need RAG?

Look for these signals. The more of them apply, the stronger the case:

• Volume. Hundreds or thousands of documents, far more than any prompt can hold.
• Change. Content is updated weekly or monthly, so a fixed copy goes stale.
• Repeated questions. The same questions about policies, procedures or product details reach a few experienced people again and again.
• Search that fails. People know the answer exists somewhere but cannot find it with keyword search because they do not know the exact wording.
• Permissions. Different teams, sites or roles may see different documents.
• Accountability. Answers must point to a source a person can open and verify.

A simple test: list the 20 questions your team asks most often and note where each answer lives. If the answers are spread across many documents and systems, and finding them takes real time, RAG is worth assessing.

A worked example

The numbers below are illustrative arithmetic, not results from any real project.

A customer service team of 25 people each spends about 40 minutes a day looking up policies, product details and previous replies. That is 1,000 minutes, or roughly 16.7 hours, a day. If a RAG assistant cuts lookup time by half, the team gets back about 8 hours a day, before netting off the time spent checking answers. Whether that is worth building depends on the full cost picture, which our article on AI project ROI covers.

When is RAG the wrong tool?

RAG is not the answer to every knowledge problem. Choose something else when:

• The content is small and stable. A ten-page handbook can often go straight into the prompt.
• The answer is a live number. Stock, order status and balances belong in the system of record. Query it directly, with permissions.
• You need a consistent style or format, not new facts. That is a prompting or fine-tuning question.
• The documents are wrong or contradictory. Retrieval will faithfully surface the contradiction. Fix the content first.
• People only need to find files. Better search and tagging may be enough.

Many useful systems combine these. An assistant can retrieve the returns policy with RAG and check the customer’s order status with a direct query in the same answer.

What makes a RAG knowledge base accurate?

Most accuracy problems come from the content and the retrieval, not the model. Check these:

• Document quality. Remove duplicates and outdated versions, and mark one owner per document. If two policies disagree, the assistant cannot know which is right.
• Sensible splitting. Split by section and keep headings with their passages. A passage cut mid-table loses its meaning.
• Metadata. Store date, owner, department and document type, so the system can prefer the latest version or filter by site.
• Hybrid search. Combine meaning-based and keyword search, so product codes, names and clause numbers are matched exactly.
• Reranking. A second pass that reorders candidate passages by relevance usually improves answers noticeably.
• Grounded answers. Instruct the model to answer only from the retrieved passages, cite them and say when the answer is not there.

How do you handle permissions and sensitive documents?

Access control must be enforced at retrieval, before anything reaches the model. If a person cannot open a document in the intranet, the assistant must not retrieve passages from it for them. Copy each document’s access rights into the index and filter every search by the asking user’s identity.

Also decide early which content stays out entirely, such as personnel files or legal correspondence, and how personal data is handled under GDPR. This is a general point, not legal advice: check your obligations with your own advisers. Where the system runs matters too. It can run in your own cloud tenancy on Azure, Google Cloud or AWS, which many organisations prefer for sensitive content.

How do you test a RAG system before rollout?

Build a test set before you build the assistant:

1. Collect real questions. 100 to 200 from tickets, emails and chat history, including awkward ones.
2. Record the right answer and its source for each, agreed with the document owners.
3. Measure retrieval. For each question, did the correct passage appear among the top results?
4. Measure answers. Correct, partly correct, wrong, or correctly declined.
5. Check citations. Does each cited passage actually support the answer?
6. Fix and rerun. Most fixes are to content, splitting or search settings, not the model.

Keep the test set and run it after every content or configuration change. When people report a wrong answer, add that question to the set. We treat this kind of measurement as part of the build, not an afterthought.

How should you start with RAG this month?

1. Pick one collection. One department’s policies, one product line’s documentation or one team’s procedures.
2. Name an owner who can confirm which documents are current.
3. Clean it. Remove duplicates and outdated versions.
4. Write the test set from real questions.
5. Run a small pilot with a handful of users, citations on, and a simple way to flag wrong answers.
6. Measure retrieval hit rate, answer accuracy and time saved, then decide whether to extend it.

Key takeaways

• RAG grounds a model’s answers in passages retrieved from your own documents, with sources, without retraining the model.
• A knowledge base needs RAG when content is large, changing, permissioned and must produce checkable answers.
• Live figures belong in direct system queries, and small, stable content can often go straight into the prompt.
• Accuracy depends mostly on clean content, sensible splitting, hybrid search and grounded, cited answers.
• Enforce access rights at retrieval and test against real questions before and after every change.

If you are weighing up whether your knowledge base needs RAG, talk to us.