Trask

This is a very poor article. It manages to both be demeaning to people who want to learn something new while also being riddled with factual errors.

Here are just a few factual errors:

1. The 11-line example is the simplest possible neural network with a very basic implementation of gradient descent in a high-level vectorization framework (numpy). It's exactly what you'd cover in the first week of a course on neural nets. But obviously there are many things to learn beyond that before you can solve real problems effectively with neural nets. In fact, the example code is 100% lifted from Hacker News user williamtrask's great article [1] which ironically closes with a list of 10 topics to investigate next to start learning the additional things you'd need to make your neural net useful for real problems.

2. This section:

> '“Deep Learning” and n-Layers of depth is just a neural network that runs its output through itself. This is called Recursive Neural Networks (RNN), because you loop the loop.'

..is full of errors. RNNs are Recurrent, not Recursive. They have nothing to do with having more layers, but with having a memory of previous states. And in deep learning, having more layers is often a strategy to reduce the dimensionality of the input and learn higher-level representations of data by using a variety of tricks/layer types/connections. It's not just "looping the loop".

3. The whole section about not initializing the random state and thus getting different results…

They are, but once you start learning about them, you realize the "black magic" part comes mostly from their mathematical nature and very little from them being "inteligent computers".

A neural net is a graph, in which a subset of nodes are "inputs" (that's where the net gets information), some are outputs, and there are other nodes which are called "hidden neurons".

The nodes are interconnected between each other in a fashion, which is called the "topology" or sometimes "architecture" of the net. For example I-H-O is a tipical feed forward net, in which I (inputs) is the input layer, H is the hidden layer and O the output layer. All the hidden neurons connect with all the input neurons "output", and all the output neurons connect to the hidden neurons "output". The connections are called "weights", and the training adjusts the weights of all the neuron with lots of cases until the desired output is achieved. There are also algorithms and criteria to stop before the net "learns too much" and looses the ability to generalize (this is called overfitting). In particular, a net with one hidden layer and one output layer is a universal function estimator -- that is, an estimator that can model any mathematical function of the form f(x1, x2, x3, ..., xn) = y.

Deep learning means you're using a feedforward net with lots of hidden layers (I think it's usually between 5 to 15 now), which apply convolution operators (hence the "convolutional" in the name), and lots of neurons (in…

Yes for sure. I am very math-averse (almost failed calculus 1, skated by pre-calc in high school... not my intelligence area) and though I don't have a full time job doing something in AI, I do feel like I can build models that solve real problems (and am doing so in an internship right now).

Probably start with Andrew Ng's Machine Learning course. It has a significant amount of Math in it-- try to understand it, but seriously do not worry about it. Just get the high level concepts, try to get some intuition on machine learning ideas and techniques. You don't need to do the assignments or work too hard on the course (but obviously it's helpful if you do).

Then read these. Don't worry if you're still confused at first. It's fine. Just go through 'em kinda slowly and try to see what's going on.
https://iamtrask.github.io/2015/07/12/basic-python-network/
http://karpathy.github.io/neuralnets/

By now if you're still into it, I highly recommend Chris Olah's blog: http://colah.github.io/
It has some pretty complicated ideas in there, but the articles are illustrated and explained very well so you can get more of a feel for Neural Networks while also getting very excited about them.

Then it's probably time to start really building things. I would use Keras (https://keras.io/) at first, because it's very easy to get cool results without understanding everything under the hood. Do a tutorial or two, it's pretty intuitive and if you've done everything above you should understand…

The below resources are the ones I used when I started to learn about DL/NNs. Some of them are focused specifically on certain applications, but I found them helpful, too.

Basic NNs:

http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/ (a whole series, all worth reading)

https://gist.github.com/sthware/c47824c116e6a61a56d9 (my code based on the above)

http://iamtrask.github.io/2015/07/27/python-network-part2/

http://rolisz.ro/2013/04/18/neural-networks-in-python/

ML:

http://cs229.stanford.edu/materials.html

http://onlinestatbook.com/2/index.html

DL in general, RNNs, RNTS, CNNs, some others:

http://cs224d.stanford.edu/syllabus.html

http://cs231n.stanford.edu/syllabus.html

http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

http://arxiv.org/pdf/cs/0205070.pdf

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf

http://alexdavies.net/talks/

http://www.socher.org/index.php/Main/SemanticCompositionalityTh
roughRecursiveMatrix-VectorSpaces

http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/

An alternate theory is that you can make use of ML to do useful tasks through an understanding of just high school math (basic algebra) and basics of python. I'm not sure if that's actually true, but I'm inclined to be in this camp as abstractions are used in virtually every other tasks. The amount of extra value that can be had from applying even basic ML techniques is so great, that there is probably a lot of upside to using ML even if they're only able to hire practitioners.

A great resource specifically tailored to those that don't have a especially strong grasp of probability and statistics is Grokking Deep Learning

https://iamtrask.github.io/2016/08/17/grokking-deep-learning/

The original Python implemation found at: http://iamtrask.github.io/2015/07/12/basic-python-network/

X =: 4 3 $ 0 0 1 0 1 1 1 0 1 1 1 1

Y =: 4 1 $ 0 0 1 1

w =: 3 1 $ 0.5069797399 0.5703818579 0.9265880619

e =: 2.718281828459045

iter =: ^:

dot =: +/ . *

T =: |:

output =: verb define

1 % 1 + e ^ 0 - X dot y

)

train =: verb define

l1 =: output y

l1_err =: Y - l1

l1_delta =: l1_err * l1 * 1 - l1

y + (T X) dot l1_delta

)

w =: (train iter 1000) w

l1 =: output w

l1