SemiAnalysis

https://semianalysis.com/

They have some of the best publicly available analysis on these topics. The full details and numbers are hidden behind the institutional accounts which are priced for investors (not something you sign up for personally) but they're generous with what they send out in their newsletter.

If you're not familiar with resources like this I could understand how you'd assume that the providers are hemorrhaging money on inference costs, because that is that story that gets parroted around spaces like Hacker News.

You could ignore all of that, though, and go check OpenRouter to see how much providers are selling high parameter count models. They're not entirely at the level of the SOTA models, but the biggest open weight models are not that far behind in complexity either. They're being sold an order of magnitude cheaper than what you pay for the APIs from the major players. We don't know exactly how big the major models are, but it's unlikely that they're more than 10X more compute intensive from the leaks we do have.

I think they will be, but more for hand-off. Local will be great for starting timers, adding things to calendar, moving files around. Basic, local tasks. But it also needs to be intelligent enough to know when to hand off to server-side model.

Android crowd has been able to run LLMs on-device since LlamaCPP first came out. But the magic is in the integration with OS. As usual there will be hype around Apple, idk, inventing the very concept of LLMs or something. But the truth is neither Apple nor Android did this; only the wee team that wrote the attention is all you need paper + the many open source/hobbyist contributors inventing creative solutions like LoRA and creating natural ecosystems for them.

That's why I find this memo so cool (and will once again repost the link): https://semianalysis.com/2023/05/04/google-we-have-no-moat-and-neither/

Odd to see this the day after the appearance of what's supposed to be a "leaked" memo from inside Google advocating the opposite course -- claiming that the outsider crowd playing with (mostly) LLaMa derivatives is getting good results, and that the best thing for Google might be to try to engage. It claims, for example, that the open-source crowd, trying to find cheaper training methods using smaller, more targeted data sets -- because they have to -- has found ones that work as well, by some metrics, as anything that the groups with huge compute clusters. And that because they work as well, with less resources, they can subsequently explore alternatives a whole lot quicker than anything involving training new base models from scratch.

The (supposed, unsourced) leak: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

Here's a sober analysis based on actual experts with access to more information than what is publicly released:

https://newsletter.semianalysis.com/p/how-to-kill-2-monopolies-with-1-tool

Skeptics welcome to review the following section:

> These are extraordinary claims and thus demand extraordinary evidence. Let’s take them one-by-one:

And here is their conclusion:

> Naysayers will point out a million reasons why this is improbable, difficult, etc. - and they are mostly correct. There is a big difference between lab-scale and industrialized, high-volume tools. Substrate itself realizes this and agrees they are in for a lot of development and scaling pain. Still, they have at least developed some impressive capabilities on the most complex part of the process (litho) in a short amount of time (2-3 years).

Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905

With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastructure

The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted.

---

The fact that they're using MoE was news to me and very interesting. I'd love to know more details about how they got that to work. Variations in that implementation would explain the fluctuations in the quality of output that people have observed.

I'm still waiting for the release of their vision model which is mentioned here but we still know little about, sans a few demos a few months ago.

> what Alibaba spends their time doing?

Not most of the time (pre-training takes a long time), but post-training is where most of the value is, yes.

Famously it is all that OpenAI did between GPT 4o and GPT 5.3 (or 5.2?) - they didn't manage to complete a pre-training run[1], and all their progress was done with post-training (!)

Post training what Cursor spends their time doing, and that has built a model that is competitive with the best coding models out there.

It isn't limited at all.

If you want to complain about something not being open source, complain about the lack of good open source RL environments (Prime Intellect excepted).

[1] https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-the