Benchmarks¶
The benchmark that uses your tools: toolcalling.ai
Static benchmarks can't tell you how a model handles your schema, your users, and your edge cases. At toolcalling.ai, two anonymous models answer the same real tool-calling task in the Arena and you vote for the better one. The blind pairwise votes build a live, real-world leaderboard.
Public benchmarks¶
Links checked 2026-11-01.
Function calling¶
| Benchmark | Measures |
|---|---|
| BFCL V4 (Berkeley Function Calling Leaderboard) · code · data · paper | The de facto standard: AST-checked single and parallel calls, multi-turn stateful calls, irrelevance/abstention, and agentic tasks (web search, memory, format sensitivity). V3 blog · V4 blog |
| Nexus Function Calling Leaderboard | Zero-shot calls to real-world APIs |
| NESTFUL | Nested call sequences where one call's output feeds the next |
| ComplexFuncBench | Multi-step, constrained calls with long context |
| ACEBench | Normal, special (incomplete or ill-formed requests), and agent cases; EN/ZH |
| When2Call | Deciding between call, ask a follow-up, or say it can't answer |
| HammerBench | Fine-grained function calling in mobile-assistant scenarios |
| Seal-Tools | Self-instruct tools, including nested calls |
| T-Eval | Step-by-step: instruct, plan, reason, retrieve, understand, review |
| API-Bank | 73 runnable APIs in dialogue: planning, retrieval, calling |
| ToolHop | Multi-hop tool use |
Agentic / multi-turn¶
| Benchmark | Measures |
|---|---|
| τ-bench · τ²-bench · τ³-bench · leaderboard | Agent + simulated user + tools + domain policies (retail, airline, telecom). Reports pass^k reliability |
| ToolSandbox | Stateful, conversational, interactive evaluation |
| ToolBench / ToolEval · StableToolBench | 16k+ real APIs, pass/win rate; StableToolBench makes it reproducible |
| TaskBench | Tool-graph task decomposition and calling |
| AgentBench | Broad agent evaluation across 8 environments |
MCP¶
| Benchmark | Measures |
|---|---|
| MCP-Atlas · leaderboard | 36 real MCP servers, 220 tools, 1,000 tasks |
| MCP-Bench | 28 live servers, ~250 tools, multi-server planning |
| MCP-Universe · site | Real servers across maps, finance, browser, repos, and more |
| MCPMark · leaderboard | Realistic create/read/update/delete workflows (Notion, GitHub, Postgres, Playwright, filesystem) |
| LiveMCPBench | Navigating a large pool of MCP tools |
General-purpose leaderboards (secondary)¶
These rank models on general chat, coding, or vision preference, not on tool calling specifically. They're useful context when choosing a base model.
| Leaderboard | Notes |
|---|---|
| LMArena (now Arena) | Large-scale blind pairwise human votes on general prompts. Formerly Chatbot Arena; renamed Arena in January 2026 (announcement) |
For tool calling, prefer the function-calling and agentic benchmarks above and the toolcalling.ai leaderboard.
Reading benchmark results¶
- Match the benchmark to your use. Single-call AST scores say little about 20-step agents, and the reverse is true too.
- Look at the sub-scores: irrelevance, multi-turn, and long context often differ far more than the overall score.
- pass^k beats pass@1 for production: users don't retry.
- Contamination: public test sets leak into training data. Keep a private eval (Evaluation).
- Format sensitivity: the same model scores differently with prompt-based vs native tool calling.
- Then validate on your own tools in the toolcalling.ai workspace, and check the community leaderboard.