Five tools, fifty tools, five hundred: selection changes with scale.
Use direct exposure for small catalogues, deterministic scoping at medium scale and retrieval plus reranking for large tool systems.
Short answer
Tool selection should become more structured as the catalogue grows. With a handful of tools, expose the complete eligible set and write precise descriptions. With dozens, filter by role, module and workflow before the model sees them. With hundreds, retrieve and rerank tool definitions, include dependencies and validate schemas before execution. The numbers are teaching ranges, not universal thresholds. Move stages only when evaluation shows confusion, latency or context cost.
Key points
- 01
Eligibility and security filtering happen before semantic tool selection.
- 02
Tool descriptions need purpose, use conditions, exclusions and examples.
- 03
Large catalogues require retrieval evaluations and executable end-to-end tests.
01 / SITUATION
A larger catalogue changes the selection problem
A model can compare five explicit tools in context. At fifty, similar names and overlapping responsibilities create confusion. At five hundred, shipping every definition is costly and makes the model responsible for catalogue discovery.
Scale should therefore change the architecture, not only the prompt. Deterministic product context narrows the domain first. Semantic methods then rank within the allowed set. The executor still validates the selected tool and arguments.
02 / MECHANISM
Three operating ranges
Treat these ranges as diagnostic prompts. Actual transitions depend on description quality, overlap and model performance.
- 01Around five: expose directly
Send all eligible tools. Use distinct names, typed arguments, clear descriptions, when-to-use and when-not-to-use guidance, plus realistic examples.
- 02Around fifty: scope deterministically
Filter by product module, role, workflow stage and current state. Group related tools and keep high-consequence writes behind explicit policy.
- 03Around five hundred: retrieve definitions
Index tool descriptions and examples, retrieve candidates, rerank them with the query and context, then expand required dependencies.
- 04Validate before execution
Check tool eligibility again, parse arguments against a schema, enforce permission and state invariants and attach provenance.
- 05Evaluate executable outcomes
Test paraphrases, near-neighbour tools, missing dependencies, denied operations, tool errors and multi-step chains. Verify resulting state, not only the selected name.
03 / In real work
Small support surface
Five tools cover status, navigation, source lookup, ticket proposal and escalation. All can be presented with strong descriptions.
Multi-module operations product
The current module and role reduce fifty tools to the six relevant to the open workflow.
Enterprise action layer
Hundreds of internal operations are indexed as capabilities. Retrieval proposes a candidate set, but the host validates each call.
Common failure modes
- Presenting every tool to the model regardless of role or workflow.
- Using vague descriptions that differ only by a noun.
- Treating semantic tool ranking as an authorisation layer.
- Selecting a tool correctly but generating invalid or unsafe arguments.
- Testing tool names without executing against controlled fixtures.
Implementation checklist
- Every tool has a typed input and output contract.
- Descriptions include purpose, exclusions and examples.
- The host computes the eligible catalogue.
- Large catalogues have retrieval and reranking traces.
- Dependencies and mutually exclusive operations are represented.
- The executor rechecks policy and state.
- Evaluations verify the final state and recovery path.
What this approach does not claim
- Catalogue size alone does not determine difficulty; overlap and description quality matter.
- Research results on API benchmarks do not establish safety in a live product.
- Retrieving tools adds another probabilistic stage that requires its own evaluation.
Sources and reference types
- 01τ-bench: Tool-Agent-User Interaction in Real-World DomainsYao et al. · 2025Research
- 02Model Context Protocol specificationMCP maintainers · 2025-11-25Standard
- 03Toolformer: Language Models Can Teach Themselves to Use ToolsSchick et al. · 2023Research
- 04ToolLLM: Facilitating Large Language Models to Master 16000+ APIsQin et al. · 2023Research
- 05ToolRet: Tool Retrieval for Large Language ModelsACL Findings · 2025Research
- 06ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool RetrievalLREC-COLING · 2024Research
Practical questions
Is MCP required for tool selection?
No. MCP can standardise discovery and invocation, but the same design applies to typed APIs and CLI tools.
Should the model plan multi-step tool chains?
Only inside explicit limits. Prefer known workflows for consequential operations and evaluate dependency, retry and partial-failure behaviour.
How often should tool descriptions change?
Version them with the underlying contract. Re-run routing and execution evaluations whenever meaning, schema or eligibility changes.