Andrew Gelman, a Columbia statistician, writes about statistical methods, flawed research, and problems in social science.
HN commenters recommend it for identifying erroneous social science studies and for its coverage of the replication crisis.
> You mean, I'm welcome to find supporting evidence for your argument? Shouldn't that be your responsibility?
Fair enough.
I had some trouble finding the original better example I had mentioned earlier ("I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright."), but while trying to find it, I stumbled on another.
Example 1: https://andrewgelman.com/2018/08/21/scandal-isnt-whats-retracted-scandal-whats-not-retracted/
A study gets published finding a large effect, large enough to cause a replication attempt: "The effort to replicate the original study was successful in everything except the creation of the PSU-level structural stigma variable."
The suspected reason for this replication failure (imputation of missing data) turns out to be wrong when the original authors have someone check their code for data analysis, which is found to contain an error.
If that code had been checked during peer review (or at least afterwards, by including it with the publication), the effort would have been less than a full-blown replication attempt.
Example 2: https://medium.com/@jamesheathers/the-grim-test-a-method-for-evaluating-published-research-9a4e5f05e870
(Which is what I was referring to earlier.)
Simply checking reported means and sample sizes for consistency revealed mathematically impossible results in 50% of tested papers.
He goes to great lengths to stress that such inconsistencies…
ignoring the automation aspect,
> Then you write exactly WHAT you are going to do with the data and then you send the proposed "routines"
this roughly matches up with the idea of "preregistration" of research, i.e. you define and share what your experimental method is going to be before you start looking at the data and performing analysis, to help guard against some unconscious or conscious decisions to adjust the method after you have observed the experimental data.
i've never heard of cos.io before but they're a top hit for me when searching for "preregistration" https://cos.io/prereg/
Andrew Gelman has written quite a lot about related topics in the past (the "replication crisis" - especially in the social sciences, "p hacking", "the garden of forking paths" http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf , https://andrewgelman.com/2017/03/09/preregistration-like-random-sampling-controlled-experimentation/ )
I quite like how Gelman theoretically frames this in his "forking paths" paper:
Consider the following testing procedures:
1. Simple classical test based on a unique test
statistic, T, which when applied to the observed
data [ y ] yields T(y).
2. Classical test pre-chosen from a set of
possible tests: thus, T(y; phi), with
preregistered phi. For example, phi might
correspond to choices of control variables in a
regression, transformations, and data coding and
excluding rules, as well as the decision of
which…
I remember reading http://andrewgelman.com/2016/11/13/more-on-my-paper-with-john-carlin-on-type-m-and-type-s-errors/
with its graph "This is what power = 0.06 looks like". So I got the point that you have to have sufficient statistical power. A useful rule of thumb is that you need a power of at least 0.8. You need to have some idea how big the effect is likely to. Perhaps from previous exploratory research, from claims of other researchers, from reasoning "well, if this is happening the way we think it is, there has to be an effect greater than x waiting to be discovered.". Then you work out how big a sample size you need to use. Then you roll up your sleeves and get down to work.
But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. You only need to think about the null hypothesis and ask yourself whether the probability of the data, given the null hypothesis, is less than 0.05.
So there is a bit of contradiction. p values don't really work unless you ensure that you have sufficient power. To do that you need a plausible effect size, to feed into your power calculation. And that is implicitly an rough approximate prior, 50:50 either null or that effect. You could just do a Bayesian update, stating how much you shifted from 50:50.
Basically, if you don't already know enough to have an arguable prior to get a Bayesian approach started, you don't know enough to do a power calculation, so…
That happens in statistics programs. However, I have a ML-heavy minor in CS, and based on the ML course contents at our CS dept I've seen, I'm not sure if the all their CS majors go through the the full canonical statistics curriculum, nor that they were intended to. At least the ML courses had quite much introductory probability and statistics as far as ML applications were concerned, so I understood the implication was they didn't assume that the students would have already done the similar stuff in statistics (though it certainly helped), and I can't remember a single mention of p-value there.
And then there's this, that even if your intro to probability course everywhere covers the classic statistics with p-values and hypothesis testing and frequentist confidence intervals and so on, you are not necessarily going to use them that much. I calculated some p-values and other tests with R for some example datasets a couple of years ago and never seen them since in coursework, everything we've done after that has been more or less fully Bayesian. The concepts are still fresh[1] in my mind mostly because I read some statistics blogs, such as Andrew Gelman's [2]. The irony is that Gelman does not exactly love frequentist framework, he just mentions its concepts often enough.
[1] or not totally forgotten
[2] http://andrewgelman.com/
It's an open secret in the news industry that while all of those longform articles, parallaxin' interactives and news apps do get great traffic, that traffic is nothing in comparison to that totally random and not even very good quick hit that went viral yesterday.
Personally, I also agree that you shouldn't worry too much about polishing content and instead just get it out there. That's what's so lovely about the blog of statistician Andrew Gelman for example [1], he just writes what comes to mind and his readers love it because they care more about his insights than about whether or not he's presenting an eloquently argued, watertight argument. It reminds me of the old days of blogging, and I think many of us miss those days.
That said, Kent Beck displays some really sloppy thinking here. "I have no clue about what the actual numbers are, and even though I could get them and run some analysis on them to determine the effects of quality vs. quantity, instead I'm just going to philosophize about how I think the world works, and justify my ramblings by saying that it's a POWER LAW, and power laws, that's like nature, dude, you can't fight it."
[1] http://andrewgelman.com/
Stan is a fantastic language/library/tool. (Disclosure, I've taken 2 classes with Andrew Gelman[1].
For those who haven't used it, the typical usage is to use Stan in combination with other languages (most commonly R). In the Stan language you define a model, specifying the data you'll get, potentially transformations to that data, then a set of parameters you want to fit, then finally a model saying how those parameters interact with each other and the data. You can also define priors for parameters. Then typically you save that model, and using R pass in the variables to a Stan call. The resulting fit object is returned to your original environment automatically.
It actually fits the work style reasonably well. All data munging happens as before, but instead of having some complicated model expression in, say, a glm() call, you have it in Stan language.
If you're interested in this, Gelman has two great books. Gelman & Hill's Hierarchical Models[2] which is applied and geared towards social science researchers, and Bayesian Data Analysis[3].
[1] http://andrewgelman.com/
[2] http://www.stat.columbia.edu/~gelman/arm/
[3] http://www.stat.columbia.edu/~gelman/book/