Alex Smola

Before reading the Blog's summary of the paper, I thought hey they're going to analyze the distributed algorithms[1][2] exclusively and give conclusions on how heterogenous hardware usgage of GPU/CPU works (best) when spread over the network/cloud. My guess was that MXNet would win, because Amazon [3][4] and benchmarks favored it[5]. Unfortunately surveys like this are too complex and out of date, when released, due to the fast progress in our field.

>> @agibsonccc What exactly would YOU, your company and your team like to see reviewed and what would you make different?

A standard benchmark, like the Open-Source, Automated Benchmark provided by the Phoronix-Test-Suite[6] or the benchmarksgame[7] are what we need. Everybody should be able to contribute easily, or even anonymously like with Steam's hardware survey[8].

The main problem with distributing operations with Caffe in example is the need to learn another system for a different programing flavor.

--
References

[1] https://en.wikipedia.org/wiki/Distributed_algorithm

[2] https://en.wikipedia.org/wiki/Category:Distributed_algorithms

[3] http://www.infoworld.com/article/3144025/cloud-computing/why-amazon-picked-mxnet-for-deep-learning.html

[4] http://www.allthingsdistributed.com/2016/11/mxnet-default-framework-deep-learning-aws.html

[5] See Page 52 - http://alex.smola.org/talks/NIPS15.pdf

[6] https://www.phoronix-test-suite.com/

[7] http://benchmarksgame.alioth.debian.org/

[8]…

Maybe my undergraduate institution failed mebut I don't feel that having taken linear algebra, multivariable calc, applied stats, and graduate probability over the summer that the PDF below contains readily accessible information to me. Does it require graduate linear algebra? Graduate multi-v? We're my courses not rigorous enough? Am just not smart enough? I graduated summa cum laude, so maybe I should be okay.

All I am saying that is that there seems to be a huge leap between having taken the prerequisites everyone says you should take and then seeing Cramer-Rao bounds on page 70.

http://alex.smola.org/drafts/thebook.pdf

Yes, on their site they post Torch CuDNN v4 as faster than Leaf [0]. Seems exciting for an early release.

Can it get much faster than something like Torch? I would think if CuDNN is doing most of the computation time it would be hard to see big improvements. Perhaps go the route of Neon and tune your GPGPU code like crazy [1, 2], or MXNet and think about distributed computing performance [3].

[0] http://autumnai.com/deep-learning-benchmarks

[1] https://github.com/soumith/convnet-benchmarks

[2] https://github.com/NervanaSystems/neon

[3] http://alex.smola.org/talks/NIPS15.pdf

There's quite a few good free ones of varying perspectives

http://metaoptimize.com/qa/questions/186/good-freely-available-textbooks-on-machine-learning

Esp:

http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage

http://www.inference.phy.cam.ac.uk/itprnn/book.html

http://alex.smola.org/drafts/thebook.pdf

___________

and MLPP errata: http://www.cs.ubc.ca/~murphyk/MLbook/errata.html

Berkley Course: http://alex.smola.org/teaching/berkeley2012/statistics.html

Scikit-Learn Docs: http://scikit-learn.org/dev/

Google Research: http://research.google.com/pubs/papers.html

Caltech Course: http://work.caltech.edu/telecourse.html

Basics in R: http://www.statmethods.net/advstats/glm.html

R packages for data science: http://blog.yhathq.com/posts/10-R-packages-I-wish-I-knew-about-earlier.html