Andrew Gelman and colleagues at Columbia write about statistics, causal inference, research misconduct, and replication problems in social science.
Commenters recommend it from a statistics perspective and cite it for critiques of bad science reporting and flawed research.
> Seventy-three independent research teams used identical cross-country survey data to test an established social science hypothesis: that more immigration will reduce public support for government provision of social policies. Instead of convergence, teams’ numerical results varied greatly, ranging from large negative to large positive effects of immigration on public support. The choices made by the research teams in designing their statistical tests explain very little of this variation: a hidden universe of uncertainty remains.
That's... surprising, but keep in mind that we're talking about "soft" social sciences here.
Measuring soft quantities like "public support for government provision of social policies in response to increased immigration" is very different from measuring hard quantities like, say, the location of a rocket in space at a point in time using earth as a frame of reference.
I'm looking forward to seeing what Andrew Gelman and his colleagues will say about it at https://statmodeling.stat.columbia.edu/ -- surely they will want to say something about it!
CITATION NEEDED — except not really, that's utter bollocks that doesn't warrant a counter.
But, you triggered me lol:
1.) The Chinese meaning of "atheist" may not mean what you think it means: a.) it means not following a major established religion, and b.) they have an Orwellian surveillance-state dictatorship government which very much opposes major established religions
2.) For many people, including in China, astrology is just something like buying a lottery ticket, or asking the Magic 8-Ball for advice — fun, but not something actually believed in[1].
3.) It's much more likely that 3% of atheists "believe in" astrology[2] — and the idea that a majority would is insane on its face
So yeah, nah.
[1]: https://www.pewresearch.org/religion/2025/05/21/3-in-10-americans-consult-astrology-tarot-cards-or-fortune-tellers/
[2]: https://statmodeling.stat.columbia.edu/2020/04/11/42552/
I interpreted the "difference which makes a difference” as the sensitivity of our understanding of the status quo relative to the research instance. In that sense, the critique would be about estimating impact on some dimension of understanding conditioned on a finding of credible difference. That would say that big bets are better research.
In other terms of critique, Gelman talks about two components (which may appear to be in tension), one referencing the craft of research, and the other in terms of apparent novelty of effect. My interpretation is that a critique of research should factor in both the character of the bet and an assessment of the evaluation. I don't think big is better, I think incisive is better, but that broader impact (news worthiness) is conditioned on credible surprise.
[0] https://statmodeling.stat.columbia.edu/2014/08/01/scientific-surprise-two-step/
> which is so obviously a quantum leap in terms of what it could do for human productivity
that's a very strong claim. and it's very likely, unfortunately, false.
we have the great classics all available for free in a multitude of formats. a lot of works slowly but surely mature into the public domain each year, yet there's no discernible productivity growth based on the availability of books.
also, this sadly goes for research papers too. sure, fuck Elsevier and the whole paper mill industry, but the limiting factor is expertise, tenured spots, academic integrity. (see the replicability crisis in psychology, in particular how the many times the whole scientific enterprise is backwards: https://statmodeling.stat.columbia.edu/2011/01/11/one_more_time_o/ and https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ )
Not my impression.
The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about.
Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions.
[1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
[2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-forking-paths-criticism-of-p-values-is-not-like-a-famous-borscht-belt-comedy-bit/
It sounds like you read Andrew Gelman's blog. You might be interested to review this post: https://statmodeling.stat.columbia.edu/2012/11/10/16808/
Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe.
Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it.
> If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them.
You're not forced to do those things well. Any scientific method can be cargo culted.