Smerity, a machine learning researcher, writes about neural networks, NLP, AI hype, web analytics and HTTPS privacy leaks.
HN links his conference paper roundups, his post on where HTTP referrers went, and a guide to what to watch in OpenAI benchmarks.
Good rule-of-thumb to always add dropout. Very hard to lower the performance this way.
Yes, we have to try a load of stuff, especially in deep learning, where the old feature engineering seems to be replaced with architecture engineering [1].
Yet, the stuff to try is usually well-known [2]. You should not be trying stuff completely at random (unless you do a random gridsearch for hyperparams [3]).
> we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes!
Hinton advises to start with a wide net, ensure that the net is able to learn the function, then add dropout, smaller layers, and/or regularization to reduce the overfit. This should avoid scenario's like the above.
Wanting to have the perfect model for a specific problem mathematically calculated may be an unrealistic demand. If we view machine learning as compression (using the shortest program possible to describe a function/distribution), per Kolmogorov Complexity, we can't compute this shortest program, and if we happen to find it, we can't know we did. The maths is for theory. Intuition is for the practical application. See intuition more as guided guessing, instead of random guessing: with experience you should know what are the flaws and benefits of the different models and architectures for particular problems. You can limit your search to sane narrow ranges.
[1] https://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html
[2] Finally, suppose you want to…
>Almost every recent A.I. advance has come from one tiny corner of the field, machine learning. Machine learning exposes a set of connected nodes, known as neural nets, to mass amounts of labeled real-world data in an attempt to give those nodes tacit knowledge. The breakthrough example was software that was able to identify cat pictures.
>So far, these neural nets have given us some great demos but mostly niche real-world applications. We don't have self-driving cars quite yet!
What's the author's threshold for "real-world applications"?
- Google's Youtube algorithm for recommendations uses neural nets[1]. So ~2 billion viewers being affected by it doesn't seem like a "niche" application.
- Google language translation uses neural net[2]
- Apple Siri voice recognition uses neural net
It doesn't seem like neural nets are analogous to the joke that "graphene is the wonder material that can do everything except escape the research lab".
In contrast, deep learning neural nets have escaped the research lab and are widely used in production systems today.
The author's blog post is recently dated August 2021 so it seems like he's not kept up-to-date on this topic since the experimental neural net winning ImageNet in 2012. Yes, that was an artificial contest but things have progressed quickly and there is real-world commercial deployment of NN trained models.
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf
[2]…
Actually it's almost impossible for a regular user to avoid using Google products any more, e.g.:
- Over 63% of the top 100k web sites embed their web bug ( http://smerity.com/cs205_ga/ )
- I have no data, but a huge proportion of web sites use Recapcha in their sign up process. It's impossible to use some domain registrars without revealing the IP address of the probable domain registrant
- I cannot host a web site without Chrome or Google DNS donating some huge percentage of data about my visitors to Google. I'm not sure the mechanism employed here, but I've set up 2 private web sites in the past year where upon having a Chrome user visit, Googlebot turned up within a day or so, despite all efforts to ensure the URL remained private
- I cannot visit Reddit and many similar sites with a cold web browser cache without revealing my use of the web site to Google, thanks to their AJAX library hosting
- A technical user has no difficulty blocking Urchin, but e-mail is a different issue entirely. I cannot find the article now, but one user reported 40% of all mail in his inbox passed through Google's servers at some point, even though he wasn't a Google user.
On the average day, whether I like it or not (and really I don't), I generate tens to hundreds of new records in Google's databases.
To expand on this a bit: You need a baseline feature representation for structure data for "traditional" problems.
What it DOES is shifts the problem to layer architectures which is a hard problem in and of itself:
What vonnik is talking about and what's already been addressed in a sibling comment was images.
"Features" are the pixels which handles generating the "baseline representation" needed for feature engineering.
Another thing of note here is it just highlights that ETL is important. Eg: image sizes, window lengths for time series etc.
Normalization is also still hugely important for many neural network problems.
Smerity says it best:
http://smerity.com/articles/2016/architectures_are_the_new_feature_engineering.html
I provided a write-up of interesting ICLR submissions, primarily for NLP, but have also provided a small snippet explaining why I thought the paper was important or how it fit with other recent relevant papers.
http://smerity.com/articles/2016/iclr_2017_submissions.html
I'd also strongly recommend people read through all the submissions themselves as every person seems to select a different set of papers depending on what they're interested in. My write-up has very few papers discussing CNNs for example as I'm focused on RNNs for the most part.
http://openreview.net/group?id=ICLR.cc/2017/conference
https://smerity.com/articles/2016/google_nmt_arch.html section "V6: Residuals are the new hotness". You can think of each layer (or I guess a pair of layers) as trying to correct errors in the previous layer. If the previous layer is close enough, the next layer can output 0. This is more stable than a straight linear activation, where one layer might move up a bit as the next one moves down to compensate! ReLU lets the earlier layer fine-tune itself without the effect being undone by later layers.