The Kaggle Blog covers machine learning competitions, winners' write-ups, tutorials, and Kaggle's open data platform.
HN commenters point to it for competition winners' write-ups and as a place to learn data mining by doing, citing its datasets, forums, and code examples.
I agree (except for the first word), however I read the question with emphasis on "usual", as in, "What makes DNNs special?"
There's pure performance (ex., in a Kaggle competition [http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/] or on a standard data set [http://yann.lecun.com/exdb/mnist/], [http://blogs.microsoft.com/next/2015/12/10/microsoft-researchers-win-imagenet-computer-vision-challenge/#sm.001t3igmw1ctxfjwsr62l6ghxb23w] ), but that's what makes any ML method better than another.
I think the deeper awesomeness is that DNNs so good at Feature Learning from raw data. On vision, NLP, and speech problems [nice overview by Andrew Ng: https://m.youtube.com/watch?v=W15K9PegQt0] DNNs have achieved superior performance to the combination of expertly-engineered features + some usual ML algorithm.
Where a "usual ML" pipeline might look like (1) engineer features through manual effort by studying raw data and the problem domain, (2) apply ML to those features, a new DNN pipeline might look like (1) Apply DNN to raw data.
First off, removing the feature engineering step could be a huge savings in human time spent. Second, there's the potential to get a better answer (!) when you're done.
But more than that, the DNN pipeline holds the promise of more regular, systematic improvement. We (as engineers) don't have to wait for a bright idea about how to construct a feature from the data. Instead, we can focus on (1) collecting more and…
An example of what I was aiming at is to use ML methods like deep nets to detect early-onset diabetic retinopathy. Diabetic retinopathy is the leading cause of blindness and at least 90% of new cases could be reduced with early detection and proper treatment. [1]
Especially in third world countries where there is no eye doctor in sight, these cheap automated methods can be deployed on a mobile phone and achieve near-human expert level accuracy. [2]
Then organizations like Watsi can use data science and predictive modeling to reduce fraud and get both detection and treatments to those most in need. [3]
[1] https://en.wikipedia.org/wiki/Diabetic_retinopathy
[2] http://blog.kaggle.com/2015/09/09/diabetic-retinopathy-winners-interview-1st-place-ben-graham/
[3] https://dssg.uchicago.edu/
About IBM Watson, the entire thing is unfortunate, I completely agree. Their marketing department upsold IBM Watson for cancer treatment. But I know that a lot of great research scientists worked at IBM on Watson. What they were doing was legit advancing machine learning too. That's the thing about marketing: if IBM were to deploy 10.000 phones with a neural net to improve early detection of disease in a third world country, I probably won't even hear about it, and I work with ML. But for IBM Watson, everybody and their grandmother goes: That's the AI that beat Jeopardy. It's a thin line between being majorly successful in marketing and crossing the line into damaging your reputation and…
In a data mining / machine learning class I took in university a key point my teacher made was that of the 'No free lunch' theorem. In that class the theorem was offered on its face and I haven't yet read the original paper (just summaries and recounts) (http://web.archive.org/web/20140111060917/http://engr.case.edu/ray_soumya/eecs440_fall13/lack_of_a_priori_distinctions_wolpert.pdf), but the point of the theorem feels very related to the point the author makes. The thesis boils down to: 'not having seen the data, there is no universally superior algorithm'. The author's point of view seems to comport with this theorem. I haven't gone through the paper myself, so I can't argue for or against it. However, in my experience as a professional statistician that has used Random Forests, SVMs, Neural Networks, etc --- finding the right algorithm takes 'feature engineering' (http://blog.kaggle.com/2014/08/01/learning-from-the-best/) and close inspection of the data and basis function. Are the data fit well by rectangles? Then you'll want to use decision trees / random forests. Curvy? Give SVM a try. Do the data take intricate patterns? Look into boosting (fitting on residuals recursively). Feature engineering boils down to the old statistical phrase, 'live with the data.' That is, you actually need use your own mind and beliefs to organize inputs prior to feeding them into an algorithm. So in my experience, what seems to be true in Kaggle competitions, and the perspective of this…
The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3].
There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15], drawing things [16] and many more.
[1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-interview-1st-place-team-%F0%9F%9A%95/
[2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/
[3] http://arxiv.org/abs/1403.6652
[4] http://arxiv.org/abs/1502.03240
[5] http://arxiv.org/abs/1411.4734
[6] http://arxiv.org/abs/1409.0473
[7] http://arxiv.org/abs/1502.03044
[8] http://arxiv.org/abs/1410.5401
[9] http://arxiv.org/abs/1506.02216
[10] http://arxiv.org/abs/1507.04808
[11] http://arxiv.org/abs/1503.08895
[12] http://arxiv.org/abs/1308.0850
[13] http://arxiv.org/abs/1412.3409
[14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
[15] http://www.sciencedirect.com/science/article/pii/S0893608014002007
[16] https://www.youtube.com/watch?v=Zt-7MI9eKEo
Why Machine Learning is Kaggle competitions.
I used an out-of-the-box algorithm, messed around a bit,
and definitely did not make the leaderboard.
Because that is not Kaggle competitions. Nearly everyone on the leaderboard is proficient in data analysis and machine learning. They all tried that out-of-the-box algorithm for their attempt. But they did not give up so easily.
Understand the business problem
If you want to predict flight arrival times, what are
you really trying to do?
This is not different from Kaggle competitions, this is a tip for performing better in Kaggle competitions. See also the GE Flight Quest: https://www.gequest.com/c/flight Those winners used industry-standard machine learning, optimization techniques, but also creative insights and hunches, like tweaking the target labels:
"A next step is to ask, “What should I actually be predicting?”. This is an important step that is often missed by many – they just throw the raw dependent variable into their favorite algorithm and hope for the best. But sometimes you want to create a derived dependent variable. I’ll use the GE Flight Quest as an example: you don't want to predict the actual time the airplane will land; you want to predict the length of the flight; and maybe the best way to do that is to use the ratio of how long the flight actually was to how long it was originally estimated to be and then multiply that times the original estimate." - Steve Donoho -…
The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc.
XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning.
XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/ and https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced)
I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests.
It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).