Never Work in Theory

Certainly, experience is a reasonable way to evaluate ideas, but that doesn't mean the evaluation is valid. I agree that the bloodletting example was meant to undercut the assertion. I also think the assertion - judging only on experience - is incorrect, and that the example is a valid one. We have many examples where experience lead to incorrect conclusions.

By experience, Aristotle argued that things in motion come to rest. Everyone knew that ... until Galileo, further codified by Newton. One's experience can be wrong. Hence the reason for software usability research. See "Making Software", edited by Greg Wilson, for lay results of some of that research. Here's an example of a more direct, empirical study: http://neverworkintheory.org/2011/10/24/an-empirical-comparison-of-the-accuracy-rates-of-novices-using-the-quorum-perl-and-randomo-programming-languages.html .

You write "human beings rely on experience at all times to evaluate." We also use comparison testing, as with that paper link I gave you. A hope is to minimize preconceptions based on one's experience. We see this in the medical field all the time, where we find that even single blind testing can unduly affect the results.

We also use prediction to evaluate. Einstein predicted a certain bending of light around the sun, which wasn't based on experience.

Therefore, I place a weak meaning to "human beings rely on experience at all times to evaluate." We use it to guide our understanding, but also use other…

(Note: I don't mean for this comment to be perceived as bragging or showmanship. It's just that... well, you asked about my background.)

Computers have fascinated me since before I can remember. I spend most of my waking hours in front of them. I've been honing my craft for over 20 years. I've written and maintained projects in C, C++, C#, Java, JavaScript (both browser and server-side)[1], Perl, PHP (we all make mistakes), Python, Ruby, and a couple Lisps. In my travels, I've discovered and reported bugs in popular software such as Firefox, Chrome, Node.js, Apple's XNU kernel, and libxml2.

I completely agree that some programmers are like machine guns, firing off vast quantities of poorly-aimed code. I try my best to avoid that. I hate sloppy code. I hate repetitive code. Most of all, I hate re-inventing the wheel. If a decent library exists, I'll use it. I have no qualms with something Not Invented Here.

I pair sometimes. I do code reviews often. And I use as many profiling, testing, and static analysis tools as I can get my hands on.

It sounds like your ordeal made you a better coder. Those sorts of experiences are indispensable, but I've found it takes more to keep improving. It's very useful to become an expert on programming, not just an expert at programming. There's a growing body of literature to aid anyone interested. McConnell's Code Complete is still great. Michael Feathers has a book called Working Effectively with Legacy Code. It contains some great…

Here are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway.

* Blogs with a math focus:

http://blog.sigfpe.com

http://bit-player.org

http://mathr.co.uk/blog/

https://www.johndcook.com/blog/

* Blogs with a programming languages focus:

https://existentialtype.wordpress.com

https://bluishcoder.co.nz

http://blog.ezyang.com

http://akkartik.name

http://lambda-the-ultimate.org

http://mrale.ph

* Blogs with a UI design focus:

http://www.brandonwalkin.com/blog/

http://ignorethecode.net/blog/

http://interuserface.net

* Blogs with a graphics / 3D rendering focus:

http://c0de517e.blogspot.com

http://diaryofagraphicsprogrammer.blogspot.com

https://directtovideo.wordpress.com

http://graphicrants.blogspot.com

http://www.reedbeta.com

http://www.realtimerendering.com/blog/

https://blog.demofox.org

http://eelpi.gotdns.org/blog.wiki.html

* Other interesting / more general programming blogs (many of these have a "systems" focus):

http://danluu.com

http://ewontfix.com

http://chneukirchen.org/blog/

http://neverworkintheory.org

http://apenwarr.ca/log/

https://www.snellman.net/blog/

https://www.pvk.ca

http://psy-lob-saw.blogspot.com

https://randomascii.wordpress.com

http://fastcompression.blogspot.com

https://fgiesen.wordpress.com

http://blog.plover.com

http://chneukirchen.org/trivium/

http://rachelbythebay.com/w/

I saw a cool study recently (summarized well here[1]) with an empirical experiment on how well code coverage predicts how well a test suite catches bugs. They found that the number of test cases correlated well with the test suite's effectiveness, but, when controlling for the number of tests, code coverage didn't.

It was a pretty thorough study:

> Our study is the largest to date in the literature: we generated 31,000 test suites for five systems consisting of up to 724,000 lines of source code. We measured the statement coverage, decision coverage, and modified condition coverage of these suites and used mutation testing to evaluate their fault detection effectiveness. We found that there is a low to moderate correlation between coverage and effectiveness when the number of test cases in the suite is controlled for.

Given their data, their conclusion seems pretty plausible:

> Our results suggest that coverage, while useful for identifying under-tested parts of a program, should not be used as a quality target because it is not a good indicator of test suite effectiveness.

That's certainly how I approach testing: I value having a thorough test suite, but I do not treat coverage as a target or use it as a requirement for other people working on the same project.

[1]: https://neverworkintheory.org/2021/09/24/coverage-is-not-strongly-correlated-with-test-suite-effectiveness.html

Oh man, it is really funny that he ends it with telling people to read the study.

To clarify the linked study is attempting to replicate https://dl.acm.org/citation.cfm?id=1070834, THE seminal study in Test Driven Development. Well to be more precise it was replicating an existing replication of that study which failed to replicate the original results. They were trying to modify the design so as to account for issues in the experimental design that may have led to the replicated study being inconclusive.

This is significant because if you were not aware of the failed replication, and believed that TDD was supported scientifically as more productive because of that original study, then you SHOULD be reconsidering its place in your development process. If that isn't the case your opinion is unchanged by these particular results(even in the article inspiring this one the author admits that their opinion was already based on a much more thorough analysis, see: http://neverworkintheory.org/2016/10/05/test-driven-development.html).

Now what I want to know is why people insist on writing articles in this awful conversation format. It wastes a lot of words to make a simple argument poorly.

Greg Wilson indeed is tremendously helpful in facilitating "the industry" to think about our craft:

https://github.com/gvwilson

edit: wow, in his project "It will never work in theory" he's fairly sober about the ability of "the industry" to reflect on "the craft"

https://neverworkintheory.org/

> about the project:

> People have been building complex software for over sixty years, but until recently, only a handful of researchers had studied how it was actually done. Many people had opinions—often very strong ones—but most of these were based on personal anecdotes or the kind of "it's obvious" reasoning that led Aristotle to conclude that heavy objects fall faster than light ones.

in the 2024 retrospective:

> Conclusion

> The comedian W.C. Fields once said, “If at first you don’t succeed, try, try again. Then quit. There’s no point in being a damn fool about it.” Thirteen years after our first post, it is clear that our attempts to bridge the gulf between research and practice haven’t worked. We look forward to hearing what actionable plans others have that will find real support from both communities.