Agentic RAG vs RAG comes down to who controls retrieval. Standard retrieval-augmented generation (RAG) retrieves passages once, using the question as the query, and the model answers from them [2]. Agentic RAG lets the model plan its searches, judge what comes back and search again before it answers [19, 26]. On multi-hop questions, where each lookup depends on an earlier one, interleaving retrieval with reasoning raised retrieval by up to 21 points and answers by up to 15 on four datasets [6]. The loop also adds model calls, time and new ways to fail [13, 16, 24].
A survey of agentic RAG, revised in April 2026, treats it as one design among several. For simple fact retrieval and well-scoped queries, it finds modular pipelines sufficient, with lower overhead [19]. This explainer traces the idea through the primary papers and sets out when the extra cost buys accuracy.
Agentic RAG vs RAG: what changes
| Aspect | Standard RAG | Agentic RAG |
|---|---|---|
| Query | The question, once [2] | Written by the model at each step from what it has found [6, 16] |
| Retrieval calls | One per question [2, 7] | Several, until an answer or a step cap [5, 6] |
| Retrieved passages | Used as returned [9] | Graded, with a rewrite or a fallback when they miss [12, 29] |
| Whether to retrieve | Always [9] | Decided by the model or a router [9, 13] |
| Control flow | A predefined code path [26] | Directed by the model [26] |
A model developer draws the same line between workflows, where models and tools follow "predefined code paths", and agents, which "dynamically direct their own processes and tool usage" [26]. A fixed retrieve-then-answer pipeline is a workflow. A model that picks its next search from what it has read is an agent.
Graph RAG is a different axis. It changes the index, building an entity graph and summaries for questions about a whole corpus [18], and the survey treats agentic RAG over a graph as a category of its own [19].
Why one retrieval pass misses multi-part questions
Multi-hop questions need facts from more than one document [1], and in sets built to resist shortcuts one step "critically relies on information from another" [3]. A query written from the question alone cannot ask for a fact the question does not name. In the words of IRCoT's authors, "what to retrieve depends on what has already been derived" [6].
Models also fail to combine facts they hold. In one family of models, the share of questions answered wrongly despite correct answers to every sub-question stayed at about 40% across sizes [4]. An engineering study of three RAG systems lists an "incomplete" failure, where the answer leaves out information that was in the context, and for a question about several documents its remedy is to ask about each separately [11]. The same list includes missing content, which no loop can fix [11].
Interleaving search with reasoning
ReAct (2022) alternated reasoning with search, lookup and finish actions over a simple Wikipedia interface [5]. Grounding cut hallucination: none of ReAct's sampled failures were hallucinations, against 56% of chain-of-thought failures [5]. Search became the weak point. Search result errors made up 23% of failures, and the model at times repeated its previous thoughts and actions [5]. With a 540B-parameter model, ReAct alone scored 27.4 exact match on HotpotQA against 29.4 for chain of thought, and combining ReAct with self-consistent chain of thought reached 35.1 [5].
IRCoT (ACL 2023) uses each new reasoning sentence as the next query, and stops at an answer or after 8 steps [6]. It used plain BM25 keyword search throughout, so the gains came from better queries. A 3B model with interleaved retrieval outperformed one-step retrieval with a model 58 times larger [6].
On FRAMES, a set of 824 multi-hop questions, one commercial model rose from 0.41 accuracy without retrieval to 0.66 after five rounds of its own searches [16]. With every needed article in the prompt it reached 0.73 [16]. It needed example queries and an instruction not to repeat them. Without those it "goes in the wrong direction in search retrievals and never corrects itself" [16]. Since 2025 the search behaviour is also trained: Search-R1 reports gains of 41% and 20% over RAG baselines for a 7B and a 3B open model [20].
Deciding when to search and grading results
Agentic designs also decide whether to retrieve at all, and check what came back.
- Low confidence. FLARE retrieves when the sentence it is about to write contains low-confidence tokens [7].
- Reflection tokens. Self-RAG (ICLR 2024) trains a model to decide when to retrieve, to label whether each passage is relevant and how well each statement is supported [9].
- A separate grader. Corrective RAG scores retrieved documents with a small fine-tuned model and turns to web search when they fail [12].
- Routing by complexity. Adaptive-RAG sends each question to no retrieval, one step or multi-step retrieval [13].
- Rewrite and retry. An agent framework's tutorial grades the retrieved documents and rewrites the question when they miss [29].
The right setting depends on the task. In Self-RAG, retrieving less cost little on one dataset and more on another [9].
Many searches at once
Questions whose parts do not depend on each other can be searched in parallel. A model developer reported in June 2025 that its research system's lead agent starts 3 to 5 subagents in parallel, each calling tools in parallel, and that these changes cut research time by up to 90% for complex queries (vendor-reported) [27]. One cloud search service's documentation, last updated 17 September 2026, describes query planning that splits a complex query into subqueries and runs them in parallel, in a preview version of its API [28]. Our explainer on when a multi-agent system beats a single agent covers when splitting work helps and when it hurts.
Retrieval quality still sets the ceiling
"Agentic reasoning cannot compensate for consistently poor retrieval," the survey's authors write [19]. In IRCoT, retrieval on one dataset improved by 21 points with no gain in answers for the 175B-parameter model [6]. In a model developer's own tests, adding chunk context, keyword search and reranking cut the share of relevant chunks missing from the top 20 results from 5.7% to 1.9% [25].
More passages are not free. Beyond 20 retrieved documents, accuracy improved by only 1.5% and 1% for two commercial chat models [8]. For many long-context models, quality rose and then fell as passages were added, which the authors traced to near-miss "hard negatives" [17].
When the corpus fits in the prompt
A model developer advises that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt with no need for RAG [25]. Long contexts fail in their own way. Models used facts at the start or end of the context better than facts in the middle [8], and in 2024 tests only half of 17 models that claimed 32K tokens or more held up at 32K [14]. Long context also takes GPU memory, since the KV cache grows with context, as our explainer on GPU memory for LLM inference shows.
Li et al. found long context ahead of RAG on average when resourced sufficiently, with RAG at a much lower cost. Letting the model answer from retrieved chunks first, and using the full context only when it declined, cut cost by 65% and 39% for two commercial models [15].
What the loop costs, and when to stop
Every round is another model call. Averaged over six datasets with one commercial model, multi-step retrieval scored 50.87 F1 against 46.99 for one retrieval step, at 3.33 times the time per query [13]. FRAMES needed six model calls that could not run in parallel [16]. A model developer reports that agents use about 4 times the tokens of a chat, and multi-agent systems about 15 times (vendor-reported) [27].
Stopping is its own problem. One trained search agent could have answered correctly without searching in 27.7% of its search steps [23]. In a 2026 study of one commercial reasoning model, accuracy on answerable questions rose sharply with the first search and peaked around seven searches, while the model grew worse at saying an answer was absent and tokens per correct answer kept rising [24]. The papers set their own caps, 7 and 5 steps in ReAct and 8 in IRCoT [5, 6], and the survey advises "explicit stopping criteria" [19].
More retrieval also means more citations to check. Self-RAG labels whether each statement is supported by its passage [9], and our note on checking LLM citations against their sources sets out a claim-by-claim check.
When to use agentic RAG
Start with retrieval: keyword and embedding search, reranking and a check that the answer is in the corpus at all [11, 25]. Put the whole corpus in the prompt when it is small [25]. Add the loop for multi-hop questions over a large corpus, with a step cap and a router that keeps simple questions on one pass [6, 13, 19].
In the agent systems Rillor builds, specialist agents retrieve from documents, databases and APIs, every result is logged as evidence, and an audit step checks each citation against that evidence before an answer goes out.
Read the full report
The full report adds the IRCoT, FRAMES, Self-RAG and Adaptive-RAG tables, the seven failure points of single-pass RAG, the long-context comparisons, the stopping studies and all 29 sources.