Sean Goedecke

Re productivity: the METR study is seriously flawed overall, and:

1. if you disaggregate the highly aggregated data, it shows that the slowdown was highly dependent on task type, and tasks that required using documentation or novel tasks were possibly sped up, whereas ones the developers were very experienced with were slowed down, which actually matched the developers' own reports

2. developers were asked to estimate time beforehand per-task, but estimate whether they were sped up or slowed down only once, afterwards, so you're not really measuring the same thing

3. There were no rules about which AI to use, how to use it, or how much to use it, so it's hard to draw a clear conclusion

4. Most participants didn't have much experience with the AI tools they used (just prompting chatbots), and the one that did had a big productivity boost

5. It isn't an RCT.

See [1] for all.

The Anthropic study was using a task far too short to really measure productivity (30 mins), and furthermore the AI users were using chatbots, and spent the vast majority of their time manually retyping AI outputs, and if you ignore that time, AI users were 25% faster[2], so the study was not a good study to judge productivity, and the way people quote it is deeply misleading.

Re learning: the Anthropic study shows that how you use AI massively changes whether you learn and how well you learn; some of the best scoring subjects in that study were ones who had the AI do the work for them, but then…

This is a great explainer from an LLM perspective, and it would be interesting to see a computational scheduling explanation in depth. I presume that hyperscale LLM companies extensively examine the computation trace to identify bottlenecks and idle bubbles, and develop load balancers, pipeline architectures and schedulers in order to optimise their workload.

The batching requirement for efficiency makes high security applications quite difficult, because the normal technique of isolating unrelated queries would become very expensive. The nVidia vGPU GPU virtualisation time shares GPU memory, and every switch requires unload/reload context switches, doubtful they have deduplication. Multi-Instance GPU (MIG) splits GPU memory between users, but it is a fixed partitioning scheme (you have to reboot the GPU to change), and nobody wants to split their 96GB GPU into 4x24GB GPUs.

Makes me wonder what the tradeoff is for putting second level memory on the GPU board (i.e. normal DRAM), so that different matrix data can be loaded in faster than over PCIe, i.e. the HBM becomes a cache.

(I'm also really liking the honesty in the authors book on Software Engineering, not in the dry IEEE sense, but as a survival guide in a large enterprise. https://www.seangoedecke.com/book/ )

> I still believe those who rave about it are not writing anything I would consider "engineering".

Correct. In fact, this is the entire reason for the disconnect, where it seems like half the people here think LLMs are the best thing ever and the other half are confused about where the value is in these slop generators.

The key difference is (despite everyone calling themselves an SWE nowadays) there's a difference between a "programmer" and an "engineer". Looking at OP, exactly zero of his screenshotted apps are what I would consider "engineering". Literally everything in there has been done over and over to the death. Engineering is.. novel, for lack of a better word.

See also: https://www.seangoedecke.com/pure-and-impure-engineering/

For those here who are dismissive of the value of the humanities, consider that no problem and no solution is purely technical; there are always "humanistic" aspects. One can - and many do! - ignore these, or even be totally unaware of them, but they're there to be understood all the same.

If you're curious what I mean by this, Sean Goedecke's post "How I Ship Projects At Big Tech Companies" [1] is a superb example, particularly his definition of "what does it mean to ship?" No idea whether he's somebody who would say "the humanities are important" but I don't think you can understand his thesis as a technical one.

[1] https://www.seangoedecke.com/how-to-ship/

There was a blog post the other day arguing that great developers aren't 10x but infinity-x. It sounds ridiculous, but their point was that great developers don't simply program faster but solve problems other developers are entirely unable to solve. And that makes sense to me. It's the difference between an AI boot camp graduate and someone with an ML PhD. They might apply a sklearn function at the same speed, but as soon as complex problems appear, the former hits an unsurpassable wall.

edit: https://www.seangoedecke.com/weak-engineers/

>Shipping is a social construct within a company.

Thanks for flagging this, this was an epiphany for me today, so for anyone else struck by it I'm linking directly to the article it's from (same author, and linked from the article in the context the parent mentions, just not linked directly in their post above):

"How I ship projects at big tech companies" https://www.seangoedecke.com/how-to-ship/

Also the HN comments on it from when it was originally posted: https://news.ycombinator.com/item?id=42111031