Retraction Watch

I remember some of the same overhyped news stories the science journalist who wrote the article submitted here remembers. I especially remember the breathless (and false) reports about a "gene for" this or that human behavioral trait. The science news cycle[1] frustrates journalists, because every new study with an incremental finding (which may not even be replicable) has to be hyped up by research organization press offices, in the interest of obtaining more funding.

The author's follow-up on a famous science story from early in his career is thought-provoking. Indeed, editors are more nervous about publishing stories, even very well reported stories, that question good news and expose hype or even fraud than editors are about publishing stories on the latest science hero.

On the whole, it's good news that more and more scientists and journalists are alert to the possibility that a preliminary research finding may be false and overhyped besides. Here on Hacker News, we can keep one another alert by remembering the signs to look for whenever we read a new research finding news story.[2]

Hacker News readers who want to learn more about how research articles become retracted may enjoy reading the group blog Retraction Watch[3] compiled by two veteran science journalists with lots of help from tipsters in the science community. I think I learned about Retraction Watch from someone else's comment here on HN.

[1] http://www.phdcomics.com/comics/archive.php?comicid=1174

[2]…

IMHO, ARPA-H and other new agencies need to change how funding is allocated.

Right now, in experimental sciences, lots of professors from prominent universities have a monopolistic attitude towards research. They have just become middlemen. They manage large groups, have become completely detached from technical things and use their position to secure most funds in the field and also veto others from publishing or getting funded. This is a recipe for disaster, stagnation and fiascos as we can see e.g. in the field of Alzheimer's disease.

Innovative laboratories like LMB Cambridge or Cold Spring Harbor, where a great deal of work was done during the golden age of molecular biology, were at odds with this approach. All funding was internal, and the number of people managed by senior researchers was capped to a single digit.

In short, we need funding policies that weed out scammers and promote good technical work. ARPA for example is quite aware of this problem, and IMHO has much better policies than, say, the European Research Council (ERC). ERC is quite infamous as they don't even bother about withdrawing funds from laboratories that are known to have manipulated research results and had to retract their publications (produced using ERC's funding!). https://retractionwatch.com and https://forbetterscience.com are full of these stories.

I don’t think 50% or greater is a good standard for defining what we would consider to be a major problem with scientific fraud. I’m sure you’re well aware that “major” can refer to relative impact depending on the context of the issue being discussed. In the case of scientific publishing, I would argue that even 5% fraud rate would be grounds for labeling this as a major problem.

That said, this issue has received a lot of attention in recent years. Some articles try to put a positive spin on it but the stark truth of where we are today is that there are no standards enforced requiring data publishing (“available on request” doesn’t cut it) and reproducibility. This, combined with the amount of money in this industry and the impact that it has on public policy is an acceptable situation.

Some good coverage:

https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review

(Lots of links to follow on this above post)

https://retractionwatch.com/

https://www.science.org/content/article/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty

(This one puts a positive spin on the editorial process but the fact remains that there is a massive flood of fraudulent papers being published. Going back to standards of open data plus verified reproducibility would go a long way to mitigate this problem.

I like Phil but around the table this morning with a few working/publishing scientists they all disagreed with his assertion that this paper has done more harm than good.

Consensus was that this would lead to more people interested in the field and what actually does work.

There’s heaps of sloppy science out there. There are massive structural issues in how science is done.

There’s obviously not enough money or prestige in condensed matter physics if Phil thinks this is a bad hoax and it’s bad for Science.

Within the space of a month this was resolved. It wasn’t even published. Go to pharma, medicine, vet, ag and you will see hoaxes that last years. Reviewers who don’t have any relevant knowledge. Journals which won’t retract until you threaten to sue them. Universities that will take no disciplinary action against hoaxers at all. LK-99 was almost debunked in a single media cycle.

The people who have taken this to reduce the credibility of science rather than these fallible humans who succumbed to their impulse for fame didn’t give science any credibility in the first place.

EDIT: shout out to our favourite website retraction watch. Anything you read there remember, that’s science working and some Scientist somewhere who likes being right has vanquished their enemy in the academy. https://retractionwatch.com/

Elisabeth Bik has done some amazing work catching these kinds of errors. I occasionally follow Retraction Watch which cites her efforts regularly, [0] about 500 times or so. However, as she points out, she doesn't get compensated for this work:

> We need this to be a career that people can make money and use their talents in. [1]

Journals, IMO, should adopt something like a Knuth reward check [2] as an actual system. They, after all, are the ones profiting the most from our present system of scientific publications. I'm not sure what's fair; maybe 100 USD for catching a major post-publication error?

It's currently a system where most of the incentives operate in opposition to self-correction.

[0] https://retractionwatch.com/

[1] https://retractionwatch.com/2019/05/07/meet-elisabeth-bik-who-finds-problematic-images-in-scientific-papers-for-free/

[2] https://en.wikipedia.org/wiki/Knuth_reward_check

> They aren't shortcuts for the lazy, they're shortcuts for non-scientists who aren't capable of fully evaluating the science alone.

It's a shortcut fraught with potential for deception, as even a casual glance through a site like Retraction Watch will demonstrate:

http://retractionwatch.com/

I'm not sure what you mean by "evaluating the science." A scientific paper should present a hypothesis, the author's best attempt to disprove the hypothesis, and an interpretation of the evidence gathered in the processes of testing the hypothesis. There's going to be a back-story, and it's likely to be quite involved.

The article does a good job of presenting a method for navigating a paper on this basis. I don't see what checking credentials adds to the process. On the contrary, it may do harm.