James Heathers, a former academic, writes about research integrity, flawed statistics in published studies, and problems in peer review and science publishing.
HN links his posts on leaving academia and "In Mice" claims. One commenter calls him a bit dickish, citing his long-hair post.
There's kind of a lot of issues. Especially with psychology and sociology and the areas they intersect.
1. Replicability of social psychology results are estimated at 25%. (Cited in article, from http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248)
2. People are not reporting non-significant results. Which means we get an incomplete picture of the validity of certain hypotheses. (from article)
3. The numbers don't line up when comparing observed power and significance of results. (from article)
4. Some results data is erroneous or falsified. The GRIM test that surfaced a while back shows numeric "results" that cannot possibly be derived from the sample data. (https://medium.com/@jamesheathers/the-grim-test-a-method-for-evaluating-published-research-9a4e5f05e870) (https://www.vox.com/science-and-health/2016/9/30/13077658/statcheck-psychology-replication)
5. Many authors on the papers don't want to share the data from their papers - even if it's a condition of publication. (from a previous article about the GRIM test that I can't find) This doesn't directly equate to maliciousness or deceit, but it illustrates incorrect attitudes in the field. Some authors consider it undignified to have someone "check their work". Others are concerned of fallout if their data contains errors. (Generic mainstream blog/press article on reproducibility https://www.vox.com/2016/3/14/11219446/psychology-replication-crisis)
6. Ridiculously small sample…
> You mean, I'm welcome to find supporting evidence for your argument? Shouldn't that be your responsibility?
Fair enough.
I had some trouble finding the original better example I had mentioned earlier ("I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright."), but while trying to find it, I stumbled on another.
Example 1: https://andrewgelman.com/2018/08/21/scandal-isnt-whats-retracted-scandal-whats-not-retracted/
A study gets published finding a large effect, large enough to cause a replication attempt: "The effort to replicate the original study was successful in everything except the creation of the PSU-level structural stigma variable."
The suspected reason for this replication failure (imputation of missing data) turns out to be wrong when the original authors have someone check their code for data analysis, which is found to contain an error.
If that code had been checked during peer review (or at least afterwards, by including it with the publication), the effort would have been less than a full-blown replication attempt.
Example 2: https://medium.com/@jamesheathers/the-grim-test-a-method-for-evaluating-published-research-9a4e5f05e870
(Which is what I was referring to earlier.)
Simply checking reported means and sample sizes for consistency revealed mathematically impossible results in 50% of tested papers.
He goes to great lengths to stress that such inconsistencies…
You definitely don't want to come across as worked up or vitriolic. I would note that the guy who wrote this article is sometimes a tiny bit dickish though, check this article out:
https://medium.com/@jamesheathers/long-hair-dont-care-5eeba266ec52
Sample comments: If that title leads you to expect off-brand evo psych exploring massive behavioural changes induced through the complicated medium of hairdressing… well, you’d be right.
Mmmm.
It's also worth bearing in mind your audience.
This article was written by two guys who by and large are going after junk science in psychology, nutrition, and so on, where the worst motivation can only be to advance the careers of the scientists - and quite possibly the errors are just the result of over-optimism or lack of statistical abilities.
For that sort of work it makes sense to try and fix psychology from within, by pointing out errors. There isn't much point in trying to spread the word through the wider world that psychology is full of errors: what's the best outcome, exactly? A few people stop being invited to give TED talks? Big deal. Psychology isn't going to stop being studied, nor should it be!
But in the case of economics, there are two problems:
1) Economics is absolutely riddled with not just bad individual papers but entire sub-fields that are nonsense, it's an epidemic. It probably cannot be fixed by shaming individuals who make simple errors via their own journals and hoping for the field to improve. Even when…
> arXiv only supports a handful of related disciplines, whereas Elsevier has literally thousands of journals in scores of disciplines
Yes, you should probably scale the operating costs of arXiv according to the number of papers published. arXiv's reported overall cost is a few dollars for each paper hosted. Compare this with Elsevier's subscription fees or most journal's $1000-plus fees to put your article in "author pays" open access.
> Someone still has to drive the blinded reviews
Yes, an editor, which in my field is essentially always an unpaid volunteer.
> Don't forget either that you have to keep a legal team on-hand to deal with how you handle all this free information
This should probably apply to arXiv to. Can you show me evidence that arXiv is having lots of legal trouble just because they are distributing PDFs? The only legal trouble I could see them getting into is essentially trolling from Elsevier et al about "their" papers getting uploaded.
> If you (royal 'you') think that navigating academic politics is so easy
I do not think this.
I think we agree that, as you say, "the technical part is not the difficult part", but I disagree with you in that you seem to imply that large for-profit publishers are somehow necessary (see your earlier message, "from technical details to user workflows to reputation/trust to administrative politics to scientific politics to sourcing verification/reviews to a hundred more things I can't think of off the top of my…
Hmm, wait a moment.
59% of white kids moved up.
43% of white kids moved down.
Even accounting for rounding, there's no way to get those numbers. It's even worse in other places:
25% of black kids had parents who grew up in the bottom two levels and moved up at least one.
78% of black kids had parents who grew up in the top three levels and moved down at least one.
Those two groups should sum to less than 100%, certainly not 103%.
Also, except for the "Moving to Opportunity" experiment, most of this seems to ignore the distinction between correlation and causation.
This reminds me of the recent finding that 50% of papers in reputable psych journals reported mathematically impossible data. https://medium.com/@jamesheathers/the-grim-test-a-method-for-evaluating-published-research-9a4e5f05e870#.61n0pjt6s The more people start to dig in, the more the "social sciences" look unscientific.
Since few people will find it, the response on the web page is
https://medium.com/@jamesheathers/a-lot-of-people-are-hung-up-the-fact-that-its-possible-to-collect-ages-in-finer-grains-to-the-c2739f9e0b9b#.9gc1sesfz
> A lot of people are hung up the fact that it’s possible to collect ages in finer grains — to the nearest month, or day, etc.
> Remember a) this is just a hypothetical illustration and b) the vast majority of the time, age is collected exactly how I’ve described here.