RAG Foundations and the Shift to Agentic Retrieval

Retrieval-augmented generation remains necessary because a general-purpose language model does not automatically know an organization's private information. Pre-training a model on every private document, database, file share, and operational record is not a practical way to keep knowledge current. Large context windows do not remove this problem either. Even when a model can accept a very large prompt, placing all available information into that prompt is expensive, difficult to secure, and unreliable. The model may overlook the most relevant passage, mix together similar records, or answer from general knowledge instead of the intended evidence.

RAG separates knowledge access from answer generation. The system first retrieves evidence, then supplies that evidence to a model, and finally asks the model to produce an answer grounded in what was found. This means that the quality of the answer depends on a chain of decisions that happens before generation:

1. Which source should be searched?
2. Which retrieval method fits that source?
3. How was the source parsed and represented?
4. Which documents or records were selected?
5. Which chunks, rows, or results were placed in context?
6. Was enough surrounding information preserved?
7. Did the model receive clear instructions about how to use the evidence?
8. Did the system require a final synthesis after tool use?

A weak result may therefore be caused by the model, but it may also be caused by bad parsing, poor chunk boundaries, an incompatible embedding model, overrestrictive metadata, an incorrect SQL query, a missing database permission, or a tool loop that stopped before producing an answer. Mastering RAG requires understanding the whole evidence pathway.

Naive RAG is usually described as a fixed sequence. Documents are split into chunks. Each chunk is converted into an embedding. The embeddings are stored in a vector database. A user question is embedded, the nearest chunks are retrieved, and a language model generates an answer from those chunks. This pattern is useful, but it assumes that semantic vector similarity is the right way to retrieve every kind of information.

That assumption does not hold in a mixed knowledge environment. Different data types need different retrieval methods:

• Unstructured documents can use semantic vector search, keyword search, metadata filtering, and reranking.
• Structured business data is often better queried with SQL than converted into document chunks.
• Graph-shaped information needs graph-oriented queries rather than ordinary vector similarity.
• File systems may be explored through command-line or file-search operations.
• Current or external information may require a web-search tool.
• A long document may require full-document analysis after a smaller search identifies the correct file.

The system should therefore select the retrieval mechanism according to the source and the question. A product-manual question may start with hybrid document search. A sales question may use SQL. A current-information question may use web search. A broad request to summarize a complete manual may begin with chunk retrieval to find the right document and then delegate the full file to a separate analysis agent.

Agentic RAG adds a reasoning and orchestration layer to this retrieval process. Instead of running one predefined search, the main agent can form a retrieval strategy, call one or more tools, inspect intermediate results, refine the query, and continue until it has enough evidence. Tool access may be exposed through ordinary function calls, MCP servers, packaged agent skills, code execution, or sandboxed environments. The important change is not the packaging mechanism. The important change is that retrieval becomes a controlled multi-step process.

A typical agentic retrieval loop can work as follows:

1. Interpret the user's actual information need.
2. Decide which sources are relevant.
3. Select the first retrieval tool.
4. Generate a query or set of tool arguments.
5. Inspect the returned evidence.
6. Decide whether the evidence is sufficient.
7. Run another search, apply a filter, query a database, or delegate deeper analysis if needed.
8. Stop retrieval and synthesize a direct answer.

This is a shift from "retrieve once and generate" to "plan, retrieve, inspect, refine, and answer." The shift is valuable because real questions are often underspecified. A user may give a product code without naming the document. The system may need one search to locate the manual, another to identify the relevant section, and a full-document analysis to understand a constraint that is distributed across several pages.

Agentic behavior also creates new risks. An agent can choose the wrong tool, overuse metadata filters, repeat searches, reach a tool limit without answering, or attempt a query that exceeds its permissions. More autonomy therefore requires more controls, not fewer controls. The system needs:

• A narrow set of tools with explicit responsibilities.
• Database and storage permissions that remain secure even when the model behaves incorrectly.
• A configurable maximum number of tool rounds.
• A stop mechanism for the user.
• A rule that forces final synthesis when the last model action is only a tool call.
• Tracing that shows every prompt, tool call, argument, result, and model response.
• Tests that confirm the agent cannot access data outside its intended scope.

Retrieval engineering and context engineering are the two disciplines that make this work.

Retrieval engineering covers how information is found. It includes parsing, chunking, embeddings, keyword search, metadata, reciprocal rank fusion, reranking, SQL generation, web search, and document-level analysis.

Context engineering covers what the model sees at each step. It includes the system prompt, conversation history, retrieved chunks, tool descriptions, tool results, subagent outputs, and the amount of prior interaction retained. The purpose is not to maximize context. It is to supply the smallest useful body of evidence and instructions that is sufficient for the current task.

This distinction explains why full-document analysis should often be isolated. Loading a long manual into the main agent's context can crowd out the conversation, tool history, and final reasoning. A subagent can receive the whole document in a separate context, perform a focused task, and return only the result. The main agent remains responsible for coordinating retrieval and answering the user.

A useful mental model is to treat RAG as an evidence operating system. The language model is one component. The system also needs source selection, permissions, retrieval, ranking, context assembly, orchestration, tracing, persistence, and validation. The visible answer is the final product of all of these hidden mechanisms.

Practical operating principles:

• Do not assume that a larger model will compensate for weak retrieval.
• Do not assume that a larger context window makes retrieval unnecessary.
• Match the retrieval method to the data type.
• Keep the main agent focused on coordination and final synthesis.
• Delegate large or specialized analysis to isolated workers.
• Make every retrieval action observable.
• Enforce permissions below the model and application-prompt layer.
• Add agentic behavior only after the ingestion and search foundations are reliable.