FastML

It's literally indistinguishable from random static: http://silverspaceship.com/static/shot_2.png My guess is it may be making some small gains by learning "pixels are 20% more likely to be black than white" or something like that.

Dictionary wouldn't work either, unless you think the NN is more likely than random to produce sequences of bits that are exactly the same.

There may be regularities in deep NNs. For example, many learned features are basically exactly the same but rotated or translated in a different way. This is because NN's aren't inherently invariant to rotations or translations, so they have to learn multiple features for every possible variation. Another possible regularity is that the weights tend to cluster on specific sections of the input image and ignore the rest.

But this entirely depends how the weights are projected into an image. The image provided looks like random static. The standard method is to just create a separate image for every feature, and for every pixel becomes the weights of the NN from that input pixel. Like this: http://fastml.com/images/deep-learning-made-easy/layer2visualization.png

Standard image compression software should be able to work with that. But this won't work in higher layers, because there are only 3 to 4 color channels in an image.

In fact, since the first layer in that example is just gabor filters, you could manually code a program to create gabor filter units for that layer, and get massive compression. Don't…

I think this is a great thing to put out into the community, and I hope it gains widespread adoption.

I do have a few things to comment on here:

The architectures themselves have never been closed source (hence publication). Frankly, I think Alex didn't release his implementation because until TITAN came along, you needed two GTX580 GPUs to do distributed training. This is unique (and difficult) to setup and support! Just "putting the code out there" is not useful to many researchers, and the architecture is in the paper already to implement and extend or modify. You will be bombed to death with support requests, even if you just say "this is provided as is". Many researchers don't want to deal with that IMO.

Data is EVERYTHING for training these architectures, and the ImageNet dataset itself is still semi-closed source, and hard to get. You have to sign up to get their license coverage if you want to download the original images for research. I think you can get weblinks to the original data, which you could then ostensibly crawl yourself, but that gets murky from a copyright perspective.

It will also take a very, very long time to wget all 1.2TB of ImageNet. My download took about 45 days, so keep that in mind!

Most people just do not have the hardware to train a network of this magnitude. CIFAR10 is a much more appropriate dataset for consumer grade hardware/non lab-funded work IMO. I think this is one of the reasons most advances on this front have simply released…

From Yann LeCunn.
---

Numenta, Vicarious, NuPic, HTM, CLA, etc.

Jeff Hawkins has the right intuition and the right philosophy. Some of us have had similar ideas for several decades. Certainly, we all agree that AI systems of the future will be hierarchical (it’s the very idea of deep learning) and will use temporal prediction.

But the difficulty is to instantiate these concepts and reduce them to practice. Another difficulty is grounding them on sound mathematical principles (is this algorithm minimizing an objective function?).

I think Jeff Hawkins, Dileep George and others greatly underestimated the difficulty of reducing these conceptual ideas to practice.

As far as I can tell, HTM has not been demonstrated to get anywhere close to state of the art on any serious task.

I still think [Vicarious] is mostly hype. There is no public information about the underlying technology. The principals don’t have a particularly good track record of success. And the only demo is way behind what you can do with “plain vanilla” convolutional nets (see this Google blog post and this ICLR 2014 paper and this video of the ICLR talk).

HTM, NuPIC, and Numenta received a lot more publicity than they deserved because of the Internet millionaire / Silocon Valley celebrity status of Jeff Hawkins.

But I haven’t seen any result that would give substance to the hype.

Don’t get fooled by people who claim to have a solution to Artificial General Intelligence, who claim to have AI systems that…

There was a recent Reddit AMA with Yann LeCun's [1] where he comments on Torch7[slightly edited]:

Torch is a numerical/scientific computing extension of LuaJIT with an ML/neural net library on top. Torch7 is what is being used for deep learning R&D at NYU, at Facebook AI Research, at Deep Mind, and at Google Brain.

(At Facebook) We are using Torch7 for many projects (as does Deep Mind and several groups at Google) and will be contributing to the public version. We are using Torch for most of our research projects (and some of our development projects) at Facebook. Deep Mind is also using Torch in a big way (largely because my former student and Torch-co-maintainer Koray Kavukcuoglu sold them on it). Since the Deep Mind acquisition, folks in the Google Brain group in Mountain View have also started to use it.

Facebook, NYU, and Google/Deep Mind all have custom CUDA back-ends for fast/parallel convolutional network training. Some of this code is not (yet) part of the public distribution.

The huge advantage of LuaJIT over Python is that it way, way faster, leaner, simpler, and that interfacing C/C++/CUDA code to it is incredibly easy and fast.

Torch is maintained by Ronan Collobert (IDIAP), Koray Kavukcuoglu (Deep Mind. former s=PhD student of mine) and Clément Farabet (running his own startup. Also a former PhD student of mine). We have used Torch as the main research platform in my NYU lab for quite a while.

You could say that Torch is the direct heir of Lush, though…

This book is really coming together. It's been a while since I've put together a (100% not comprehensive) list of good places to start if you're looking to learn more and/or use deep learning in your projects.

Open source

Pylearn2 (used to win kaggle galaxies competition): http://deeplearning.net/software/pylearn2/

Theano (symbolic math library used by Pylearn2): http://deeplearning.net/software/theano/

Deep learning tutorials with theano (build your own neural networks): http://www.deeplearning.net/tutorial/

Demos

Convnet JS: http://cs.stanford.edu/people/karpathy/convnetjs/

Sentiment Analysis: http://nlp.stanford.edu:8080/sentiment/rntnDemo.html

3d word cloud (webgl): http://wordcloud.ersatz1.com/

Commercial

Ersatz (I'm a co-founder, it's a PaaS providing neural network software with cloud GPU servers): http://www.ersatzlabs.com

Good Reading

Deep learning of representations: looking forward http://arxiv.org/pdf/1305.0445v2.pdf

Zero-Shot Learning Through Cross-Modal Transfer: http://arxiv.org/pdf/1301.3666v2.pdf <-- C'mon, that's pretty amazing...

Solution for the Galaxy Zoo challenge: http://benanne.github.io/2014/04/05/galaxy-zoo.html

Pylearn2 in practice: http://fastml.com/pylearn2-in-practice/

If I had a 760 I'd likely just run all my experiments right on my device. As it stands my only machine is a Early 2009 macbook pro with a Geforce 9600M. The nice part about pylearn2, and theano is that the symbolic expressions compiled by theano can run on your processor too, albeit much slower. You can always test to make sure 1 to 2 epochs work locally before sending it off to AWS for GPU computation.

I'd be very willing to buy a 760, or even a 770 at their current prices. The only thing holding me back is that i'd have to buy an entire computer in which to place the card. Haha :D

If you're interested in just how fast video cards can be for deep learning as compared to CPUs take a peek at the results on [0]. That is for older model GPUs, and they're an order of magnitude faster. Though as I understand it the Geforece 5xx cards are superior for scientific computing as compared to the 6xx and 7xx series which are more gaming oriented. (May still outperform 5xx due to raw speed at the cost of some additional CPU time). Have a look at the appendix on [1] for more info on Fermi vs Kepler GPUs.

[0] http://deeplearning.net/tutorial/lenet.html#running-the-code

[1] http://fastml.com/running-things-on-a-gpu/