Plain-English explanation
An LLM's knowledge is baked into its trained weights at the time it was built; it can't natively look things up. RAG bridges that gap: instead of relying purely on what the model already "knows," the app running it first searches a separate store of documents or facts for anything relevant to the current message, then hands the model both the user's message and that retrieved material together, so the model can generate a response grounded in specific, current, or private information rather than only its training data.
How it works
RAG was introduced as a general technique by Lewis et al. in a 2020 paper that combined a pre-trained language model with a retrieval system over a large document index, showing that grounding generation in retrieved passages produced more specific and factual output than generation from the model's parameters alone. A typical RAG pipeline has two stages:
- Retrieval. The user's message (or the recent conversation) is converted into an embedding (a numerical representation of its meaning) and compared against a pre-built index of stored content to find the most relevant matches, typically via vector similarity search.
- Augmented generation. The retrieved passages are inserted into the model's context window alongside the user's message, and the model generates its response with that material directly available to reference.
The model itself is not being retrained or permanently updated by this process: RAG changes what the model sees at a given moment, not what it has learned.
Adult-AI use
RAG is one of the main techniques companion apps use to implement long-term memory: rather than trying to cram an entire relationship history into the context window of every message, the app stores past conversation details in a searchable index and retrieves only what's relevant to what you're currently saying, then feeds that back to the model. A good retrieval can recall a specific detail from weeks earlier without requiring an enormous context window or the entire chat history. A poor retrieval can return irrelevant or outdated information and make the character's "memory" inconsistent.
How it differs from long-term memory and character memory
RAG is a specific technical method; long-term memory and character memory are the product-level outcomes it's often used to achieve. Not all long-term memory is built with RAG (some apps use simpler fixed-profile or summarization approaches instead) and RAG itself isn't exclusive to memory features; it's also used more broadly for grounding a model in reference documents, knowledge bases, or up-to-date information generally.
Limitations and misconceptions
- RAG reduces but doesn't eliminate hallucination. A model can still ignore, misread, or contradict retrieved material, grounding in retrieved text lowers the risk of fabrication but is not a guarantee against it.
- Retrieval quality bounds output quality. If the retrieval step fetches the wrong or irrelevant passages, the model's response is only as good as what it was given, regardless of how capable the model itself is.
- It's not the same as fine-tuning. RAG supplies information at the moment of generation; it doesn't change the model's underlying weights the way fine-tuning does.