Data Colada

Again, academia/research is not my profession. But, some cool efforts in this area include osf.io, which is trying to be the Arxiv or Github of preregistration for scientific studies.

The best preregistration plans will typically include a declared sample or population to observe (http://datacolada.org/64), or at least clear cut criteria for which participants or observations you will exclude.

I think for the type of economics/finance research I’m most familiar with, you often implicitly announce your sample when securing funding for a research proposal. E.g. if I’m trying to see if pursuing a momentum strategy with S&P 500 stocks is profitable (a la AQR’s work), it’s pretty obvious what the sample ought to be. This is partly why that meta study I linked to earlier was able to sniff out potential signs of p-hacking.

> There's been a few studies where identical resumes have been sent out with "white-sounding" and "non-white-sounding" names, with the white names getting more follow-up.

IIRC, that study finding didn't survive the replication crisis - the most recent resume studies didn't find bias against "non-white-sounding" names.

An optimistic interpretation of the discrepancy is that the earlier studies had picked low-class "black-sounding" names. It's hard to do this kind of study cleanly since names don't merely signify race, they also signify other things like socioeconomic status and age so to do it right you'd need to carefully validate that the specific names used don't differ in implications other than race; I don't think that has really been done yet.

(more here: http://datacolada.org/51 )

Reading many scientific studies that are done poorly, I am not convinced that you need a lot of formal training in a field to identify bad science. Many frauds and errors are blatantly visible in their data, which should have a given distribution but doesn't, or in their flawed analyses. Many others have several clearly uncontrolled variables (hint: if it's not in the paper, they didn't control it).

People of a given field will also often circle the wagons around bad actors to protect their own jobs and reputations. This introduces a very strong bias in the people who should be the most highly-informed on their own subject, and sadly makes them less trustworthy than people like Data Colada (https://datacolada.org/) who go beyond their own fields.

I haven't followed this case but I found the blog posts doing the initial investigation quite interesting. Not only due to all the errors in the data but also because it gives some insight into how Excel works behind the scenes:

https://datacolada.org/109

https://datacolada.org/110

https://datacolada.org/111

https://datacolada.org/112

https://datacolada.org/114

I'm confused by her suing and her argument in court though. Is her argument that they didn't catch her in front of the computer so someone else might have fauxed the data or is her argument that no data was altered? Because it seems undeniable that the data was altered to align with the hypothesis.

The two linked source posts are excellent:

http://datacolada.org/109

http://datacolada.org/98

This latest one has great sleuthing examples:

http://datacolada.org/112

Their analysis suggested that the data (originally collected using Qualtrics) had been modified, i.e. that the data in the data files would not match the original data (still stored on Qualtrics' servers).

It turned out they were right:

We have received confirmation, from outside of Harvard, that Harvard's investigators did look at the original Qualtrics data file and that the data had been modified.