Last Word on Nothing

> If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it.

Actually, there is! It's been referred to as "the decline effect", and it's fascinating.

The New Yorker published an article about it a few years ago (http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off?currentPage=all). It has resurfaced in popularity a couple of times. It was written by disgraced journalist Jonah Lehrer, but later independent investigation failed to find any fault with the article he wrote (http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nature-of-truth/ -- worth reading because it includes statements from scientists on Lehrer's article).

So far, my favorite explanation is the direct, obvious one, which is that we're sometimes -- perhaps more often than we'd expect -- falling victim to statistics. Given the very large number of experiments and studies being conducted everywhere, and the selection for positive results, statistical anomalies are being accidentally selected for and then not being extensively enough re-evaluated much later.

The article discusses that as a possible explanation, but the journalist does his job, igniting reader interest, by suggesting the effect somehow "defies the laws of statistics". I favor the simpler explanation, which is that our meat brains simply underestimate the vast number of ways in which statistics would like to bugger us.

Some of the things that…

For those that faintly recognize the author's name... this piece was written by Jonah Lehrer, whose writings were later discredited for a combination of plagiarism, inaccuracy, and methodological weakness. Interestingly, in the ensuing reassessment of Lehrer's writings, this article has survived basically unscathed, but I think this discussion (including some statements from the interviewed scientists) is very useful:

http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nature-of-truth/