Off the Convex Path is a machine learning theory blog by academic researchers, including Sanjeev Arora, on deep learning, optimization, and generalization.
Commenters call its generalization article the best they saw recently, and one says its backpropagation post made the topic crystal clear.
You're correct. Boosting was directly formulated in the PAC framework.
(Source: http://l2r.cs.uiuc.edu/Teaching/CS446-17/LectureNotesNew/boosting/main.pdf "The original boosting algorithm was proposed as an answer to a theoretical
question in PAC learning [The Strength of Weak Learnability; Schapire, 89.]")
It took a while, but there's been a lot of work lately explaining neural nets' performance over the last 5 years of so, from papers showing PAC learnability for specific architectures (https://arxiv.org/abs/1710.10174) to work saying that most local optima are close to global optima (http://www.offconvex.org/2016/03/22/saddlepoints/), to work saying that the optimization error incurred (as separate from approximation and estimation errors) serve as a form of regularization for deep neural networks.
And understanding how these things work helps improve and speed up these methods and models: it's hybrid algorithms which are enabling performance in time-series data and more complex tasks. The future will nearly certainly use neural networks as part of many algorithms, but I doubt that the full machinery will be simple feed-forward nets of ever-increasing sizes.
Here is a fun one:
https://www.cs.cmu.edu/~tom/pubs/science2008.pdf -- "Predicting Human Brain Activity Associated with the Meanings of Nouns"
"We present a computational model that predicts the functional magnetic resonance imaging (fMRI) neural activation associated with words for which fMRI data are not yet available. This model is trained with a combination of data from a trillion-word text corpus and observed fMRI data associated with viewing several dozen concrete nouns. Once trained, the model predicts fMRI activation for thousands of other concrete nouns in the text corpus,with highly significant accuracies over the 60 nouns for which we currently have fMRI data."
This is a frequently cited paper - I stumbled across it recently as a drive-by mention in this blog post about word embeddings: http://www.offconvex.org/2015/12/12/word-embeddings-1/
Anyway - totally tangential to the topic of the OP. But maybe some interesting food for thought.
I meant in the sense of - you have benchmarkers and trainers. If you publicize your evaluation, trainers may likely have their models 'consume' it, even if only indirectly: another person creating their own benchmark from scratch may be influenced by yours, even if the new question sets are clean-room. That, and the rule of thumb that benchmark value dissipates like sqrt(age) [0]
So there is a definite advantage to never publicizing your internal benchmark. But then, no one else can replicate your findings. You should assume that the space of benchmarks that are actually decent at evaluating model performance is much larger and most of the good ones, the ones that were costliest to produce, are hidden, and might not even correspond very well with the public ones. And that the public expensive benchmarks are selective and have a bias towards marketing purposes.
[0] https://www.offconvex.org/2021/04/07/ripvanwinkle/
Well, the thing that word2vec can (and should)
be understood in terms of word coincidences (and pointwise mutual information) is important, but hardly new. I tried to explain it here:
http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html.
There is a temptation to use just the word pair counts, skipping SVD, but it won't yield in the best results. Creating vectors not only compresses data, but also finds general patterns. This compression is super important for less frequent words (otherwise we get a lot of overfitting). See "Why do low dimensional embeddings work better than high-dimensional ones?" from http://www.offconvex.org/2016/02/14/word-embeddings-2/.
Thanks for your blog post!
Are you familiar with http://www.offconvex.org/2016/07/10/embeddingspolysemy/?
I read a blog post maybe a year ago that explained word embeddings under an 'atom interpretation', in which word embeddings are really sparse combinations of 'atoms' (I don't really remember more than that). It was very interesting, but then I forgot about it. Trying to find the post again I only came up with the above paper, which is probably the same idea, but it wasn't what I originally read. Wish I could find it.
Yes.
https://joanbruna.github.io/MathsDL-spring18/
On Tanh and ReLus, the paper on SeLUs is pretty intense.
ReLUs help with vanishing gradients to some degree on a larger degree of loss and activation functions, btw.
edit: On your question about a billion data points and predicting behavior, we are getting there.
http://www.offconvex.org/2017/03/30/GANs2/