Simply Statistics

Speaking as an analytics architect ...

You'll be a lot better off spending your mental energy thinking about the outcomes you want to achieve (user engagement, upselling, growth, etc) and the types of analysis you'll need to understand what changes you need to make to produce those outcomes. Protip: this is actually really hard, and people underestimate it by orders of magnitude. A blog post by Roger Peng (with indirect commentary from John Tukey) ... https://simplystatistics.org/2019/04/17/tukey-design-thinking-and-better-questions/

One other immediate tip is to start thinking about correlating your telemetry with user surveys - again, strongly focusing on outcomes and the controllable aspects of those outcomes.

Don't let the data lead the discuisson; decide on the question you're asking, and the implications of all of the possible answers to that question (clearly yes, clearly no, mixed, etc) before you ask it.

Then engineer the lightest weight system possible to ingest, process, store, analyze, and visualize that data.

For me, that would just be:

1. Log data in whatever logging tool you like. Persist the raw stuff forever in a cheap data lake.
2. Batch at some fixed interval into a staging area of a relational DB.
3. Transform it with stored procedures for now (while you figure out what the right transforms are) into a flat fact table.
4. Visualize in Superset or PowerBI or even plain old Excel.

Once you've got the patterns of analysis at least fundamentally right…

I like this summary of the evidence that most published research is false[1]:

-"One of the hottest topics in science has two main conclusions:

Most published research is false

There is a reproducibility crisis in science

The first claim is often stated in a slightly different way: that most results of scientific experiments do not replicate."

- [Ioannidis'] "Paper: Why most published research findings are false.

Main idea: People use hypothesis testing to determine if specific scientific discoveries are significant. This significance calculation is used as a screening mechanism in the scientific literature. Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results.

Important drawback: The paper contains no real data, it is purely based on conjecture and simulation."

- then it summarises in the same way 7 other papers, including the Many Labs results.

-"I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely

We need more statistical literacy
We need more computational literacy
We need to require code be published
We need mechanisms of peer review that deal with code
We need a culture that doesn't use reproducibility as a weapon
We need increased transparency in review and evaluation of papers"

- the final paragraph:

"The Many Labs results…

Part II

With good stuff in ML you are describing a
world I've seen no solid evidence of first
hand.

Maybe that's to be expected: Long I did
well learning math in part because I
picked some really good books; due to some
lucky parts of my background, that was
easy to do. Then I tried this in
statistics and picked a junk book --
eventually figured that out.

I didn't have much respect for or progress
in statistics before I had a high end
background in probability from a star
student of E. Cinlar at Princeton from
Neveu's book, Neveu a star student of
Loeve at Berkeley. Then in stochastic
processes I got some more respect from
work of J. Doob (prof of Halmos who then
became an assistant to von Neumann, etc.)
and the material on probability in the
back of the Halmos, Measure Theory. The
Halmos work on sufficient statistics and
unbiased estimation also deserved
respect.

I've seen no such quality from ML. You
suggest that there is such quality out
there, but what I saw, e.g., Ng's on-line
lectures on ML, the idea of convolutional
neural networks for image recognition,
doesn't look nearly as solid as what I did
in probability or nearly as interesting as
some of what you point to.

E.g., just yesterday I went to

https://simplystatistics.org/

a Web site of three profs in bio-stat, and
read some of the material of one of them,
Jeff Leek, looked at the TOC and some of
the lectures in his Coursera on-line
course, some of the research problems he's
working on, some of what he…

I was very surprised when I first learned that results could be published without first having been reproduced.

A lot of psychology journals have been burned by that over the years,

http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/29045

http://simplystatistics.org/post/21326470429/replication-psychology-and-big-science

http://scholarlykitchen.sspnet.org/2012/04/26/reproducibility-an-attempt-to-test-the-psychology-literature-underscores-a-growing-fault-line/

and as a consequence, just within the last year or so, the most prestigious psychology journals are beginning to ask authors of papers on new experimental research to show replication across at least two distinct data sets. This, in turn, is driving more collaboration among researchers at different research centers. It's a long overdue development, one in accord with best practice in science,

http://norvig.com/experiment-design.html

and I hope this practice spreads to most disciplines that publish new research reports in the major journals of the discipline.

(Although the requirement for prepublication replication is not explicitly mentioned in the publication guidelines of Psychological Bulletin,

http://www.apa.org/pubs/journals/bul/index.aspx

which is the most prestigious psychology journal published in the United States, I have been told at the University of Minnesota behavior genetics "journal club" that replication across more than one data set is becoming a tacit requirement for…

Andrew Beam's post offered very persuasive evidence that Jeff Leek's intuition (deep learning yields poor performance with small sample sizes) is incorrect. The error bars and the consistent trend of higher accuracy with a properly implemented deep learning model, particularly with smaller sample sizes, is devastating to Leek's original post.

I think this is a fantastic example of the speed and self-correcting nature of science in the internet-age.

As an aside, @simplystats blocked me on Twitter, which I assume is in response to this tweet: https://twitter.com/ErickRScott/status/871586233599893505
and it seems that I'm likely not the only one blocked: https://twitter.com/jtleek/status/871693250947624961

What's most concerning about @simplystats blocking activity is the chilling effect it has on discourse between differing perspectives. I've tried to come up with a rationale for why highlighting the most recent evidence in reply to someone who sympathized with Leek's original post (btw, @thomasp85 liked the tweet) is grounds for blocking , but I can't come up with a reasonable idea.

Further aside, is irq11 Rafael Irizarry?

Update: after emailing the members of @simplystats they have removed the block on my account and offered a reasonable explanation. SimplyStats is a force for good in the world (https://simplystatistics.org/courses/) and I look forward to their future contributions.

Growth
Mr. Money Moustache [http://www.mrmoneymustache.com/]
Study Hacks, Cal Newport [http://calnewport.com/blog/]
Tim Ferriss [https://tim.blog/]
Oliver Emberton [oliveremberton.com]

Intellectual
Farnam Street, Shane Parrish [farnamstreetblog.com/blog]
Less Wrong [http://lesswrong.com/] [lesserwrong.com]
Raw Thought, Aaron Swartz [aaronsw.com/weblog]
Slate Star Codex [slatestarcodex.com]
Edge [http://edge.org/]
Melting Asphalt, Kevin Simler [http://www.meltingasphalt.com/]
Essays, Paul Graham [paulgraham.com/articles.html]
Minding Our Way, Nate Soares [http://mindingourway.com/]
Ribbonfarm, Venkatesh Rao [ribbonfarm.com]
Overcoming Bias, Robin Hanson [overcomingbias.com]
Shtetl-Optimized, Scott Aaronson [http://www.scottaaronson.com/blog/]
Raikoth, Scott Alexander [http://web.archive.org/web/20140220082152/http://raikoth.net:80/]
Heterodox Academy [https://heterodoxacademy.org/]

Technology
Hacker News [https://news.ycombinator.com/]
Worrydream, Bret Victor [http://worrydream.com/]
Naval Ravikant [https://startupboy.com/]
Unenumerated, Nick Szabo [http://unenumerated.blogspot.com/]
Stratchery [http://stratchery.com/]

Statistics
DataTau [http://www.datatau.com/]
FiveThirtyEight [fivethirtyeight.com]
Simply Statistics [https://simplystatistics.org/]
Chris Olah [colah.github.io]

Economics
Marginal Revolution [marginalrevolution.com]
Project Syndicate [project-syndicate.org]