RAG (Retrieval-Augmented Generation)

How it works

In a pure LLM without RAG, the model answers from its training data alone. For support use cases, this produces confident-sounding but wrong answers: the AI might describe a return policy that doesn't match yours, or invent a feature that doesn't exist.

RAG solves this by splitting the task into two steps. First, a semantic search (usually vector search) retrieves the three to five most relevant passages from your indexed content. Second, the LLM reads those passages and generates a response grounded in them. The retrieved passages act as a short-term memory that overrides general training knowledge.

RAG quality depends on two things: retrieval quality (does the search surface the right documents?) and LLM reasoning (does the model interpret the documents correctly?). Weak retrieval produces hallucinations even with a strong LLM.

When you'll encounter it

RAG is the architecture behind almost every enterprise AI support product shipped after 2023. When a vendor claims their chatbot 'answers from your own content' or 'reads your help center,' they are describing RAG. You'll see it referenced in technical documentation, API guides, and AI support comparisons.