The result is predictable: prompt tokens increase, confusion between similar tools grows, latency rises, and tool mis-selection becomes more common. As the model is forced to reason over an expanding set of overlapping tool definitions, both efficiency and accuracy degrade.

This scaling failure highlights a critical insight:

More tools ≠ better intelligence unless selection becomes efficient.

Prompt Bloat and the MCP Stress Test

As modern LLM agents gain access to more MCP tools, they must decide which tool to invoke for each task. The challenge is that every tool comes with a schema describing its parameters and capabilities. When dozens or hundreds of these schemas are included in the prompt, the context becomes overloaded. This prompt bloat makes it harder for the model to distinguish, recall, and choose the correct tool.

The authors compare this effect to the classic Needle-in-a-Haystack (NIAH) test, where a single relevant piece of information is buried inside a large context. As the “haystack” grows, retrieval performance drops sharply revealing limits in long-context reasoning. The same pattern emerges with tool overload: as irrelevant MCP schemas increase, the chance of selecting the correct tool decreases.

To quantify this, the paper introduces an MCP Stress Test using WebSearch tasks. For each trial, the model is given N MCP schemas (one ground-truth + N-1 distractors) and must pick and invoke the correct WebSearch tool. N varies from 1 up to 11,100, allowing the authors to measure how tool-selection accuracy, task success, latency, and prompt token usage degrade as the tool pool scales.

Why Retrieval Matters 

The RAG-MCP approach borrows a principle from retrieval-augmented generation

Don’t give the model the entire knowledge base and retrieve relevant fragments first.

Applied to tool selection, the idea becomes:

This dramatically reduces the dimensionality of the decision space. Instead of asking:

“Which tool out of 500 should I call?”

The model decides:

          “Which tool among 3 retrieved options is best?”

Example

User Query: “Find me business hotels near San Francisco airport under $200 with good reviews.”

If you dump all tool schemas into the prompt, the LLM must inspect every one, increasing

With RAG, the retriever matches semantic intent:

“hotel”, “airport”, “budget”, “reviews”

→ and returns only

 So the LLM only selects among 3 tools, not 10.

This is faster, cheaper, and less error-prone.

Inside the RAG-MCP Framework

To overcome prompt bloat, RAG-MCP applies Retrieval-Augmented Generation (RAG) principles to tool selection. Instead of flooding the LLM with all MCP descriptions, we maintain an external vector index of all available MCP metadata. At query time:

Three-Step Pipeline Diagram

Task Input → Retriever: The user’s natural-language task is encoded and submitted to the retriever. 

Retriever → MCP Selection & Validation: The retriever searches the vector index of MCP schemas, ranks candidates by semantic similarity, and optionally tests each via synthetic examples.

LLM Execution with Selected MCP: The LLM receives only the selected MCP’s schema and parameters and executes the task via the functioncalling interface

Analysis of RAG-MCP Results

The superior performance of RAG-MCP can be explained by its ability to minimize distraction and optimize the prompt for reasoning. Instead of feeding the model a large set of tool schemas, RAG-MCP filters the context and injects only the most relevant MCP definition, resulting in clearer decision boundaries and more reliable tool selection. This focused filtering dramatically reduces prompt token consumption, freeing more of the model’s context window for task-specific reasoning rather than parsing irrelevant metadata. While RAG-MCP slightly increases completion token usage compared to an ideal Actual Match scenario, this additional generation reflects deeper verification and validation steps, ultimately leading to improved accuracy and more balanced outputs.

Overall, these findings confirm that retrieval-augmented selection of MCPs effectively tames prompt bloat and enhances an LLM’s tool-selection reliability, making RAG-MCP a compelling solution for scalable external tool integration.

Final Words

RAG-MCP shows that scaling tool-enabled LLMs isn’t just a matter of adding more capabilities, it requires smarter selection. As MCP ecosystems and agent tooling grow, naively passing every tool schema to the model becomes unsustainable. Prompt bloat, distractors, and decision overload quickly erode performance, mirroring the same failure modes seen in long-context retrieval benchmarks like Needle-in-a-Haystack.

Reference : 

RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation