Yann LeCun, a deep learning researcher, hosts his papers, the MNIST handwritten digit database, and essays on his academic site.
HN links it for MNIST and his 1989 convnet paper, and commenters use it as a baseline when judging newer neural network results.
One of the best proposals for research/review system I have seen is from Yann LeCun: http://yann.lecun.com/ex/pamphlets/publishing-models.html.
The idea is that you post your paper on some common repository like arxiv. The "reviewer entity" or RE which is self-organized group with interest in sub-field can be invited to review it. The RE accepts or rejects it. Citations graph propogates to RE as reputation score.
The problem with his proposal:
1. Authors are not anonymous which can bias prestigious REs to avoid papers from unknown authors.
2. Everything still depends on citations, most of which are usually worthless as people mostly cite to fill obligatory related sections, not because they are actually using your work.
3. No obligation for REs to review anything. Doing reviews is tiresome and busy people may just avoid it unless there is a clear obligation/honor system involved.
4. Prestigious REs will be invited to review by everyone, causing highly uneven distribution for the workload.
I believe openreview system was started based on LeCun's thoughts but needless to say we haven't found a good system that can resolve above issues. More importantly, any change in system needs to come from community leaders who are part of organizing commitees for conferences or website developers like arxiv and Google Scholars. Unfortunately, last two have been virtual stasis for long time.
It's astonishing how little investment exists for our main engines of progress that is…
SVMs may make a lot of intuitive sense due to the intuitive nature of the max-margin method of building a classifier, but that doesn't make them "much, much better" than a neural network.
Neural networks are "universal" function approximators in the sense that given sufficient number of hidden layer units and training data, they can approximate any Borel measurable function. This makes them theoretically powerful. (There are issues with this "universalness" claim though when doing cognitive modelling. See "Cognition and the computational power of connectionist networks", Hadley at http://www.informaworld.com/smpp/content~db=all?content=10.1080/09540090050129745)
In practice, the best handwritten digits classifier is still neural networks. Lecun keeps scores at http://yann.lecun.com/exdb/mnist/.
Are there challenges with training neural networks? Certainly. But for many applications, there are good reasons to use a neural network. For instance, inference in a neural network is fast, and the learnt model is small in size (in feedforward networks, just a few large matrices).
SVMs on the other hand are "easier" to train, but care has to be taken to use the "right" kernel, and the learnt model in practice has to, in many cases, carry around with it up to half or more of the training data set (ie, the support vectors), making the model bigger in size, and inference slower (although there is work in making a sparser set of support vectors. See "An Effective Method of…
As an expert in both the umbrella of methods known as compressed sensing and more recently deep learning, my opinion is that these techniques are largely complementary. Deep learning is fantastic at for instance building appearance models and exploiting hierarchical nature of natural images, and hierarchical feature building in general where appropriate, while methods from compressed sensing remain as quite useful general methods via which one can efficiently extract low dimensional structure, from noisy and at times even highly corrupted datasets.
It is true that deep learning is now state of the art in certain tasks such as super-resolution for natural images, which were previously the domain of linear inverse problems, and this is due to its ability to learn useful natural image priors, something that isn't possible with simpler linear or slightly non-linear models. Meanwhile, compressed sensing style methods still excel in situations where the data does not benefit from hierarchical compression, labeled data is not available and/or the data is highly corrupted. Take for example the Netflix challenge problem, discussed in the video, for which deep learning is unlikely to offer substantial benefits, at least for the problem as stated (we just observe partial information about movie ratings). Where deep learning could potentially help in that situation is for instance grouping movies according to high level semantic information derived from text descriptions, other…
I posted this on HN back in 2023, reposting now because I don't think this article goes far enough:
I’ll make the bold claim that the following industries / companies would not exist without the USPS:
The Airline Industry: In the early days of American aviation, air transportation was unproven and not financially viable, until the USPS built the necessary infrastructure and gave contracts to airlines to allow them financial feasibility… starting in 1918! [1]
Machine Learning: In 1989 Yann LeCun wrote his seminal paper “Backpropagation Applied to Handwritten ZIP Code Recognition”, which used the USPS’s data set and has today become the hello world of machine learning tasks. More importantly this is the first commercial or industrial application of machine learning. [2]
Netflix: Before Streaming became a thing, Netflix was shipping DVDs via the USPS. The Postal Service adapted its processes and equipment to make this financially feasible, supporting Netflix through its transition to streaming. [3]
Amazon: Early Amazon was only a book vendor, the USPS offered special rates for books that made it possible for Bezos to be profitable from his garage … in 1994, thus birthing the behemoth it is today. [4]
Chickens: okay, not really. But the USPS ships millions of pounds of live chickens and other animals each year! [5]
[1] https://www.history.com/news/us-aviation-airmail-passenger-f...
[2] http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf
[3]…
err.. I think it is an evolution. Deep architectures allow for a more efficient function approximation from fewer examples than shallow architectures do.
Deep (neural) networks are really just a generalization of machine learning (on graphs). The key is that we learn the similarity function to discriminate from examples instead of specifying it a priori. You can also build classifiers other ways: linear/logistic regression based on feature vectors or by providing some similarity metric (SVM). But in these cases you are providing the discriminator function.
For example, in SVMs you have to provide a similarity measure as the "kernel" maps your feature vector into a higher dimensional space where the examples can be separated.
In deep neural networks we don't really know (or care to some extent) what the optimal feature vectors are or what the correct similarity metric is. We only care that at the end we've encoded it correctly (e.g. having chosen enough parameters/layers etc...) after training. Again the NN is some general function that applies a soft-max relationship from the inputs to outputs for each layer.
Yann LeCun has a great paper (2007) explaining this:
http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf.
I'm not sure what you are imagining the scope of your first project to be, but I would recommend you begin by understanding and implementing some well-known algorithms. Start with a Guassian Mixture model trained with the EM algorithm. Then do linear and logistic regression, perceptron is pretty simple and a simpler version of the widely used support vector machine.
The handwriting recognition database here is fantastic for testing a variety of simple ML models:
http://yann.lecun.com/exdb/mnist/
In our machine learning class, we would use data from the KDD cup for our projects. Why don't you create a submission for old KDD cups and see if your model can do better than random? 1998 is good for logistic and linear regression:
http://www.kdnuggets.com/meetings/kdd98/kdd-cup-98.html
Using the 2007 dataset, you can try out some of the matrix factoring methods that have worked well for the Netflix prize:
http://www.cs.uic.edu/~liub/Netflix-KDD-Cup-2007.html
These should all be relatively small tasks. Learning how to interpret your results and iterate your models to make them better will take longer than understanding the algorithms.