Ben Recht and collaborators write about machine learning, optimization, reinforcement learning, statistics, and research culture.
Commenters link it for deep learning and learning theory posts, a critique of the Bangladesh mask trial, and a piece on dual affiliation in AI.
I could rant about this for hours. I actually just went to a defense for a deep learning paper that had a ton of abstract algebra. I am honestly not really a fan of deep learning and algebra because all the papers to me like- seem to stop at describing some really basic feedforward network as some really specific mathematical structure but these theories a) provide very little explanation of empirical phenomena b) provide no new directions of research in terms of like useful network architectures.
I haven't really come across algebra in machine learning other than people applying it to deep learning.
i.e. https://arxiv.org/pdf/1802.03690.pdf
ie. https://icml.cc/Conferences/2018/Schedule?showEvent=2048
I don't personally find papers like this valuable but idk I have never really enjoyed abstract algebra.
For areas of mathematics to do theory in ML (and to do ML more generally!)
-probability/concentration/hoeffding bounds [the PAC model] [Key]
-linear algebra [key]
-optimization [key]
for books
-understanding machine learning by shai ben david
This book is nice since it really balances theory with a more practical understanding.
-An Introduction to Computational Learning Theory by kearns is a classic [low priority]. this is fun since the proofs are simple and deep but is very very far away from practical algorithms.
-convex optimization by boyd
Course Notes:
[I think a good alternative to blogs is stalking course notes for other schools-they are very…
I consider GPs in two ways: hyperparameter optimization (where GPs are a specific instantiation of bayesian optimization) and a powerful non-linear regression method that supports uncertainty via the rich posterior distributions you get with GPs.
However, on the side of hyperparameter optimization, an experiment comparing bayesian optimization to random search with only double the computational resources [0] suggests that BO is not at least 2x better than random search which, to me, means that in practice BO isn't my first go-to for hyperparameter optimization.
On the side of regression, its poor time complexity (you need to invert the data matrix which is O(n^3) and slow in practice with maybe >1000 points) makes it less of a first go-to for me than neural networks, which maybe irrationally benefits from its current "hotness", but also for which there are well-explored strategies for generating predictions with uncertainty [1]. Of course, GPs compare significantly more favorably on the axis of interpretability, since you can get a closed-form posterior distribution for any point of interest, and the manner in which your dataset influences this posterior is transparent.
There have been, however, interesting developments in improving the efficiency of GP regression for large datasets, such as estimating the GP with a reduced and weighted set of data points [2]. The recent Deepmind paper on neural processes is also interesting, as a combination of GPs and neural networks…
Regarding the presented studies that assessed mask wearing:
Bundgaard (not Bundagaard) 2021 [1]:
−0.3 percentage point (95% CI, −1.2 to 0.4 percentage point; P = 0.38) difference; statistically not significant. Limitations: “Inconclusive results, missing data, variable adherence, patient-reported findings on home tests, no blinding, and no assessment of whether masks could decrease disease transmission from mask wearers to others.”
Doung-ngern 2020 [2]:
11% of control not tested; 59.2% of cases linked to boxing stadium; Persons who wore masks were also more likely to practice other measures [e.g. washing hands often (79% vs. 26% for those who did not wear a mask)]
Krishnamachari 2021 [3]:
Heavily confounded. States with no mask mandate had a 2.16x higher case rate, yet “stay at home orders and school closures had no significant influence”? No analysis of “restrictions on gatherings, closing of nonessential business and restaurant closures”?
Lio 2021 [4]:
Retrospective, internet questionnaire with 24 cases.
Wang 2020 [5]:
Telephone interview; Other than that, no issues imo.
Xu 2020 [6]:
“Despite the relatively large sample size, the total cases of COVID-19 infections were still small so that the relationship between NPIs and a COVID-19 infection should be confirmed by other larger epidemiological studies. Fifth, the potential risk compensating effects of wearing a mask against other NPIs should be considered as being of a hypothesis-generating nature given the…
This article is a little too glib in my opinion, preferring citations and statements to substance and explanations.
For a more cutting and insightful critique, watch Ali Rahimi's short talk at NIPS 2017 (where he was presenting for a paper that won the "Test of time" award, for standing out in value a decade after publication). The standing ovation he received at the end indicate that his comments resonated with a significant fraction of the attendees.
https://www.youtube.com/watch?v=Qi1Yry33TQE&feature=youtu.be&t=660
Here's a teaser from the talk:
"How many of you have devised a deep neural net from scratch, architecture and all, and trained it from the ground up, and when it didn't work, felt bad about yourself, like you did something wrong? This happens to me about every three months, and let me tell you, I don't think it's you [...] I think it's gradient descent's fault. I'll illustrate..."
[Addendum]
Ben Recht and Ali Rahimi published an adendum to the talk, elaborating on the direction they envision -- http://www.argmin.net/2017/12/11/alchemy-addendum/
Ali also has a post taking a stab at organizing some puzzling basic observations about deep learning, and motivating that with analogous historical progress in optics -- http://www.argmin.net/2018/01/25/optics/
-----
PS: The first 11 minutes, on the idea of using random features (the main idea in the research he presented) are also interesting.
That makes sense. I think the author would argue that the model is never needed, just go back to the actual population, and make inferences based on random sampling. From another prior blog post[1], he states:
"So what is the remedy here? The thing is, we already know the answer: if we randomized the assignment ... it is critical to decouple the randomness used to probe a system from the randomness inherent in its system itself. Statistical models are not necessary for statistical inference, but randomness itself is amazingly… let’s say… useful for understanding natural phenomena."
This makes sense to me if there's always a population that you can random sample. And yes, you'd have to sample a lot for the sample and population summaries to converge, but this seems fine in certain contexts (i.e. when you have access to computer simulations). Your counterpoint seems correct when you aren't able to randomly sample the population in this manner.
Would you agree with the author based on the specific framing I'm making (randomly sampling population beats building a probabalistic model if you are able to do the former). Again, I'm a stats novice so apologies if I'm making an obvious point.
1. http://www.argmin.net/2021/09/21/models-are-wrong/
Yes, I am aware, I did not mean random search as in random actions, but random search with improved heuristics to find a policy.
The point being that that the bells and whistles of PPO and other relatively complaticated algorithms (e.g. Q-PROP), namely the specific clipped objective, subsampling, and a (in my experience) very difficult to tune baseline using the same objective, do not significantly improve over gradient descent.
And I think Ben Recht's arguments [0] expands on that a bit in terms of what we are actually doing with policy gradient (not using a likelihood ratio model like in PPO) but still conceptually similar enough for the argument to hold.
So I think it comes down to two questions: How much do 'modern' policy gradient models improve on REINFORCE, and how much better is REINFORCE really than random search? The answer thus far seemed to be: not that much better, and I am trying to get a sense of if this was a wrong intuition.
[0] http://www.argmin.net/2018/02/20/reinforce/