The Berkeley Gorilla research group writes about LLMs that call APIs and publishes the Berkeley Function Calling Leaderboard.
HN commenters cite its leaderboard to compare models on tool calling, noting that the best results so far come from fine-tunes and that tool calling is unsolved.
- "Are You Human" https://arxiv.org/pdf/2410.09569 is designed to be directly on target, i.e. cross cutting set of questions that are easy for humans, but challenging for LLMs, Instead of one type of visual puzzle. Much better than ARC for the purpose you're looking for.
- SimpleBench https://simple-bench.com/ (similar to above; great landing page w/scores that show human / ai gap)
- PIQA (physical question answering, i.e. "how do i get a yolk out of a water bottle", common favorite of local llm enthusiasts in /r/localllama https://paperswithcode.com/dataset/piqa
- Berkeley Function-Calling (I prefer https://gorilla.cs.berkeley.edu/leaderboard.html)
AI search googled "llm benchmarks challenging for ai easy for humans", and "language model benchmarks that humans excel at but ai struggles with", and "tasks that are easy for humans but difficult for natural language ai".
It also mentioned Moravec's Paradox is a known framing of this concept, started going down that rabbit hole because the resources were fascinating, but, had to hold back and submit this reply first. :)
"Gorilla: Large Language Model Connected with Massive APIs" (2023) https://gorilla.cs.berkeley.edu/ :
> Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke. With Gorilla, we are the first to demonstrate how to use LLMs to invoke 1,600+ (and growing) API calls accurately while reducing hallucination. We also release APIBench, the largest collection of APIs, curated and easy to be trained on! Join us, as we try to expand the largest API store and teach LLMs how to write them!
eval/:
https://github.com/ShishirPatil/gorilla/tree/main/eval
- "Gorilla: Large Language Model connected with massive APIs" (2023-05) https://news.ycombinator.com/item?id=36073241
- "Gorilla: Large Language Model Connected with APIs" (2023-06) https://news.ycombinator.com/item?id=36333290
- "Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs (github.com/gorilla-llm)" (2023-06) https://news.ycombinator.com/item?id=36524078
Hey folks, I'm on the Gemma team, we released new model(s) just recently, and I saw many questions here about function calling. We just published the docs to detail this more. In short Gemma3's prompted instruction following is quite good for the larger models and that's how you use the feature.
You don't need to take our word for it! We were waiting for an external and independent validation from the Berkeley team, and they just published their results. You can use their metrics to get a rough sense of performance, and of course try it out yourself in AIstudio or locally with your own prompts.
https://gorilla.cs.berkeley.edu/leaderboard.html
Hope you all enjoy the models!
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front.
The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.
I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.
> This is a very strange article considering that Llama, the mother of all open-weight models, has led to anything but success for Meta.
I see this as a fault with Meta's models, not a fault with the concept of open weights. The Llama family just aren't very useful. They make flowery prose but they're terrible at tool calling [1][2][3] so there just isn't much I can actually accomplish with them.
[1] https://gorilla.cs.berkeley.edu/leaderboard.html
[2] https://benchlm.ai/best/tool-use
[3] https://benchlm.ai/llm-agent-benchmarks
https://www.swebench.com
https://swe-rebench.com
https://livebench.ai/#/
https://eqbench.com/#
https://contextarena.ai/?needles=8
https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
https://artificialanalysis.ai/leaderboards/models
https://gorilla.cs.berkeley.edu/leaderboard.html
https://github.com/lechmazur/confabulations
https://dubesor.de/benchtable
https://help.kagi.com/kagi/ai/llm-benchmark.html
https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard