DX Newsletter is from a developer productivity analytics company and summarizes research on developer experience, AI coding tools and engineering metrics.
Commenters relate it to their own experience with AI coding tools needing multiple attempts, and to the difficulty of measuring software productivity.
Measuring productivity in software development, or even white collar jobs in general, let alone the specific productivity gains of even things like the introduction of digital technology and the internet at all, let alone stuff like static vs dynamic types, or the productivity difference of various user interface modalities, is notoriously extremely difficult. Why would we expect to be able to do it here?
https://en.wikipedia.org/wiki/Productivity_paradox
https://danluu.com/keyboard-v-mouse/
https://danluu.com/empirical-pl/
https://facetation.blogspot.com/2015/03/white-collar-productivity-review.html?m=1
https://newsletter.getdx.com/p/difficult-to-measure
This resonated with me based on my recent experience using Claude to help me code. I almost gave up, but re-phrased the initial request (after 7-10 failed tries) and it finally nailed it.
> 3. Performance improves with multiple attempts Allowing the o1 model 7 attempts instead of 1 nearly tripled its success rate, going from 16.5% to 46.5%. This hints that current models may have the knowledge to solve many more problems but struggle with execution on the first try.
https://newsletter.getdx.com/i/160797867/performance-improves-with-multiple-attempts