LMSYS Blog

The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this:

> *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here].

That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section:

> Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing.

> Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps:

> 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users.

> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.

> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).

> 4. Remove the model from Arena.

This sounds very reasonable to me.

When considering top tier labs that optimize inference and own the GPUs: the electricity cost of USD 5000 at a data center with 4 cents per kWh (which may be possible to arrange or beat in some counties in the US with special industrial contracts) can produce about 2 trillion tokens for the R1-0528 model using 120kW draw for the B200 NVL72 hardware and the (still to be fully optimized) sglang inference pipeline: https://lmsys.org/blog/2025-06-16-gb200-part-1/

Although 2T tokens is not unreasonable for being able to get high precision answers to challenging math questions, such a very high token number would strongly suggest there are lots of unknown techniques deployed at these labs.

If one adds the cost of GPU ownership or rental, say 2 USD/h/GPU, then the number of tokens for 5k USD shrinks dramatically to only 66B tokens, which is still high for usual techniques that try to optimize for a best single answer in the end, but perhaps plausible if the vast majority of these are intermediate thinking tokens and a lot of the value comes from LLM-based verification.

Saying that it is just in index from string prefixes into KV Cache misses all the fun, interesting, and complicated parts of it. While technically the size of the prompt-pointers is tiny compared with the data it points into, the massive scale of managing this over all users and requests and routing inside the compute cluster makes it an expensive thing to implement and tune. Also, keeping the prompt cache sufficiently responsive and storing the large KV Caches somewhere costs a lot as well in resources.

I think that the OpenAI docs are pretty useful for the API level understanding of how it can work (https://developers.openai.com/api/docs/guides/prompt-caching/). The vLLM docs (https://docs.vllm.ai/en/stable/design/prefix_caching/) and SGLang radix hashing (https://lmsys.org/blog/2024-01-17-sglang/) are useful for insights into how to implement it locally for one computer ode.

Thanks for the feedback!

We chose a set of tasks with different levels of complexity to see how this approach would scale. For LLMs, the "challenge" with NER is not the task itself but the arbitrariness of the labels in the dataset. I agree it's still much simpler than the other tasks we present (agentic RAG, agentic tool use, maze navigation).

There are definitely strong parallels to model distillation and student-teacher training, with the primary difference being that we don't simply take all the data from the larger model but rather filter the dataset based on metrics from the environment. In the "Does curation even matter?" section, we show that this generally improves the result by a good margin.

We link to Vicuna, which might be the closest reference as prior art: https://lmsys.org/blog/2023-03-30-vicuna/

Thanks!

Sigh.

Ok, so. Where to start?

Hmm… ok. Well, how abort we ignore all the rest of the stuff and cut to heart of this article:

Assertion 1) Vicuna 13B is as good as bard and chat gpt.

Assertion 2) LoRA can stack infinitely and every LoRA you stack on top is additive only, increasing capability knowledge with no downside.

I think you can make your own mind up if you choose to believe these assertions, and judge for yourself if the author is on the right track or not.

I will link however, to the actual comments by the authors of vicuna, specifically the limitations and evaluation section: https://lmsys.org/blog/2023-03-30-vicuna/#how-to-evaluate-a-chatbot

Maybe read it yourself and make an informed opinion…

(Unlike, it appears, the author of this article)

Perhaps it's not a moat.

However, if the advantage is due to things like inference infrastructure to support a massive model, that isn't easy to duplicate.

I would also say that the quality of these smaller models are good, but we also may not be measuring them correctly. Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1]. It's easy to have your inputs be inside a happy distribution for a paper but fail in the real world in ways that GPT-4 doesnt.

Lmsys would love to compare with bigger models but have limited resources. Contributions are welcome [2]

[1] https://arxiv.org/abs/2305.15717

[2] https://lmsys.org/blog/2023-05-25-leaderboard/#next-steps