Andrew Gelman

Maybe you should read something other than Casella & Berger for Bayesian statistics. The book is great, but the coverage in this topic is not. Something like Bayesian methods for hackers [1], the puppy book [2] are great tutorial style easy to read books. Big boy Gelman [2] is far more mathematical and advanced.

I highly recommend you [1] cause it's free, hands on and if you have solid stats will go fast and easy.

Anyways, the thing with Bayesian is you need to look at the problems from a different optic and learn new names for the same things you know from classic (Maximum likelihood is now called mean a posteriori and whatnot).

If I had to summarize the whole uncertainty thing I would say: Both in classic and Bayesian it holds that a function of random variables yields another random variable. Models are functions of random variables. As such, every point estimate you give has a full distribution behind. In classic this is accounted for with asymptotics and confidence intervals. In classics you have priors all the same, but using defaults and not talking about them.
In Bayesian you talk about your priors, and if you have extra information about your inference problem you include it there. Or you abuse the priors to make the model yield results that will make your boss/client happy (I worked as an economics researcher at a big bank, the whole department was Bayesian). Or choose priors that make you custom model actually compute. Or use priors that do regularization (like…

ignoring the automation aspect,

> Then you write exactly WHAT you are going to do with the data and then you send the proposed "routines"

this roughly matches up with the idea of "preregistration" of research, i.e. you define and share what your experimental method is going to be before you start looking at the data and performing analysis, to help guard against some unconscious or conscious decisions to adjust the method after you have observed the experimental data.

i've never heard of cos.io before but they're a top hit for me when searching for "preregistration" https://cos.io/prereg/

Andrew Gelman has written quite a lot about related topics in the past (the "replication crisis" - especially in the social sciences, "p hacking", "the garden of forking paths" http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf , https://andrewgelman.com/2017/03/09/preregistration-like-random-sampling-controlled-experimentation/ )

I quite like how Gelman theoretically frames this in his "forking paths" paper:

Consider the following testing procedures:

1. Simple classical test based on a unique test
statistic, T, which when applied to the observed
data [ y ] yields T(y).

2. Classical test pre-chosen from a set of
possible tests: thus, T(y; phi), with
preregistered phi. For example, phi might
correspond to choices of control variables in a
regression, transformations, and data coding and
excluding rules, as well as the decision of
which…

Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones, which researchers should be trying to do. You can't ignore Bayesian epistemology just by not using Bayesian methods. The underlying considerations in the Bayesian framework will inevitably become relevant to how you interpret your data, whether or not you use a formal Bayesian method.

The thing is, formally, frequentist methods don't tell you anything. If you get a significant p-value, that means the data you observed wouldn't often happen by chance (within your model of the null). This doesn't actually tell you if your particular hypothesis should be favored. That requires other considerations, including ones that Simonsohn is negative about in this article.

For example, Simonsohn's conclusion says:

> To use Bayes factors to test hypotheses: you need to be OK with the following two things:

> 1. Accepting the null when “the alternative” you consider, and reject, does not represent the theory of interest.

> 2. Rejecting a theory after observing an outcome that the theory predicts.

He implies that these should be points against Bayes factors. But #2 is something you actually should do sometimes. Demonstrably. In particular, if the data suggests a wildly implausible effect size, that should be a point against your theory and in favor of some more mundane explanation, like noisy data from an underpowered study…

Your comment appears to imply Bayes theorem is all you need for a scientific framework and outside of that is just inadequate mathematical know-how to deploy it. I find it amazing really, a quintessential straight-from-the-summit [1] HN comment for a couple reasons:

First, it appears to imply that such a probabilistic inductive approach to science would be free of any philosophical baggage or assumptions, when deploying Bayesianism requires an interpretation about what a probability itself is. Don't take it from me though, perhaps from Andrew Gelman, the guy who wrote the book on Bayesian data analysis [2,3].

Then, wrt the charge that those who do not use such an inductive approach (or outright rejected it, e.g. in favour of falsificationism) are bad at math. Which would include the statisticians who developed the null hypothesis significance testing framework that is still pretty dominant in science today: Jerzy Neyman, Egon Pearson, Ronald Fisher (who literally coined the term 'Bayesian') etc. There's a lot of criticism worth making about Fisher, but I'm not sure if anyone has called the guy that developed linear discriminant analysis bad at math before.

[1] https://www.smbc-comics.com/comic/2011-12-28

[2] Bayesian Data Analysis http://www.stat.columbia.edu/~gelman/book/

[3] Philosophy and the practice of Bayesian statistics http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf

> "Data is data and data is good"

This is a reasonable intuition, but is hard to square with the increased risk of publication bias for small studies (i.e. we should question whether the typical small study published is really "data" in the sense we'd like).

Since small studies are more sensitive to noise and "fishing" for an effect, and since non-results are rarely published, what happens with meta-analysis of many published small studies is that you end up primarily looking at outliers rather than typical results. Since most research questions have a directionality to what counts as "interesting", the outliers also tend to be clustered on one side rather than evenly split.

The two papers I recommend anyone read if they care about these problems are:

Why Most Published Research Findings are False

http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time

http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

This.

Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1].

Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws.

Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems.

[1] https://academic.oup.com/biomet/article/73/1/13/246001
[2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf