David Robinson, a data scientist, writes about statistics, R and the tidyverse, and data analyses of text, tweets, and trends.
HN links it for Bayesian A/B testing, empirical Bayes, tidyverse teaching, and the Trump tweet analysis, and commenters call his empirical Bayes book excellent.
Thank you for explanation. If I understand correctly, you use this approach to increase sensitivity (compared to NHST) using the same data.
Most probably, in your case, higher sensitivity (or power) comes at the cost of higher type I error rate. And this might be fine. Sometimes making more changes and faster is more important than false positives. In this case, you can just use a higher p-value threshold in the NHST framework.
You might argue that the discrete type I error does not concern you. And that the potential loss in metric value is what matters. This might be true in your setting. But in real life scenarios, in most cases, there are additional costs that are not taken into account in the proposed solution: increased complexity, more time spent on development, implementation, and maintenance.
I suggest reading this old post by David Robinson: https://varianceexplained.org/r/bayesian-ab-testing/
While the approach might fit in your setting, I don't believe most of other users of tea-tasting would benefit from it. For the moment, I must decline your kind contribution.
But you still can use tea-tasting and perform the calculations described in the whitepaper. See the guide on how to define a custom metric with a statistical test of your choice: https://tea-tasting.e10v.me/custom-metrics/
Disclaimer: arguably a quant but definitely not this kind.
In this specific case you only need access to Trump his tweets, which you can get without parsing and evaluating the firehose (expensive to get and process).
> In particular, basic NLP and sentiment analysis is pretty easy, especially like in this case when you're dealing with a very limited vocabulary.
Exactly. With Trump's tweets you can probably find easy correlations between sentiment (good or bad) and a select range of words as well as tweet times, whether he is in office or not, whether it was tweeted from his android[1] or not. Statistically speaking you can also look at his overall sentiment. Eg. simply always short the market whenever he tweets might be the profitable move 80% of the time.
[1]: http://varianceexplained.org/r/trump-tweets/
That would go against most public accounts of how Trump has been using his phone for tweeting.
For one thing, Obama's personal account (i.e. not @POTUS, but @BarackObama) explicitly says in its profile: "This account is run by Organizing for Action staff. Tweets from the President are signed -bo. [0]
No such disclaimer has been made about Trump's account, which tweets with the same characteristics and tone as it did well before his current presidential campaign [1]. You're right, though, that Trump's team does tweet for him on occasion. Either that, or Trump, who is famously not a computer user, has mastered the art of using two different mobile operating systems [2]
[0] http://imgur.com/a/ZGyzo
[1] https://github.com/sashaperigo/Trump-Tweets
[2] http://varianceexplained.org/r/trump-tweets/
I created an open source A/B test framework[0], which also uses Bayesian analysis on the dashboard. IANAS(tatistician), but from what I understand it’s still better to plan the check point in advance, rather than stop when reaching significance.
A couple of articles worth reading [1] [2] (can’t exactly vouch for their validity but seem to make some good arguments that appear thought out)
[0] https://github.com/Alephbet/gimel
[1] http://varianceexplained.org/r/bayesian-ab-testing/
[2] http://blog.analytics-toolkit.com/2017/the-bane-of-ab-testing-reaching-statistical-significance/
> For that, I might look at another source of data, Stack Overflow Careers profiles, and see which technologies tend to be used by the same developers
> http://varianceexplained.org/images/network2.jpeg
This shows "git" and "github" in a separate cluster from "C++" and "Python"? I don't understand this. These tools are used regardless of what other technologies are being used. For example, there are many Python and many C++ projects on github.
I was thinking of the same article, and found it, I think [1]? There was a higher-rated link also posted around the same time with the same post content [2]. I also found a follow up post [3] while looking for [1].
[1] http://varianceexplained.org/r/trump-tweets/
[2] https://www.r-bloggers.com/text-analysis-of-trumps-tweets-confirms-he-writes-only-the-angrier-android-half/
[3] http://varianceexplained.org/r/trump-followup/