Keras Blog

interesting topic. Could you point me to some resources so I can learn more
about it?

The following is a comprehensive introduction and survey of recent work in the
field:

https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/program_synthesis_now.pdf

Regarding novelty, neural networks are excellent at interpolation but awful at
extrapolation. What I mean by interploation is that the models learned by a
neural network represent a dense region of cartesian space circumscribed by the
neural net's training instances. The trained model can recognise and generate
instances included in this dense region, including instances not included in its
training set, by interpolation, but it is almost completely incapable of
representing instances outside this dense region by extrapolation [1].

So basically what I'm saying above is that deep learning language model code
generators can't extrapolate to programs that are not similar to the programs
they've been trained on. Hence, no real novelty. But you can go a long way
without such novelty hence my expectation that e.g. Copilot can be a nice
boilerplate generator.

______________

[1] My standard reference for this is the following article by François Chollet,
maintainer of Keras:

https://blog.keras.io/the-limitations-of-deep-learning.html

See also the second part of the series of articles where possible ways are
proposed to address this limitation:

https://blog.keras.io/the-future-of-deep-learning.html

The latter is…

I don't have much opinion on the rest of your translation, but I think this gets it wrong:

> Postmodernists make for shitty programmers

To me one of the big differences with postmodernism is a deep skepticism for grand overarching theories and perfect understandings. I think that's an immensely useful trait for a programmer.

The more I code, the more I treat any digital representation as a deeply distorted image of the reality I'm dealing with. The best I can do is have the distortions match my user's purpose. As Box wrote, "All models are wrong; some models are useful." The modernist continues to seek the perfect model. The postmodernist treats perfection as a fool's errand and ships something that works well enough.

I have seen this problem over and over in my career; a lot of architecture astronautics strikes me as rooted in modernism. But perhaps a good example that everybody can look at is early concepts of AI vs today's machine learning. When I compare Cyc [1], which has spent 30 years trying to build the seeds of perfect knowledge versus today's deep learning, that strikes me as a postmodern shift.

How does a convolutional neural network recognize a dog? We don't exactly know, not in a way that makes much sense to us. [2] But we don't care. The convnet has its own very bounded, limited version of "truth"; as long as we get a 90% match, we ship it and go out for beers.

[1] https://en.wikipedia.org/wiki/Cyc

[2]…

Keras still is the very best in terms of expressive models and end to end workflows. It leans heavily on the design idea that you should deliberately design for end to end use cases and all intermediate abstractions should be building blocks that serve precisely that purpose. This is discussed in [0] which IMO is something that deserves to be more widely talked about in software engineering. Lots of other disciplines of software engineering _say_ you should design this way, but in my experience it’s very rare no matter what discipline you’re in. Take TensorFlow itself. It’s a huge mess with no clear abstractions useful for end to end solutions. Just a hodge podge of disparate APIs and way too many underlying engineering concepts were elevated to abstractions for engineer (instead of user) convenience.

Constraining design by end to end use cases is a remarkably robust and useful process.

PyTorch is way better at having clean engineering abstractions than TensorFlow, but still falls short when things like “forward” or maintaining your own training loop and gradient metadata are necessary concepts for a practitioner’s end to end workflow.

[0]: https://blog.keras.io/user-experience-design-for-apis.html

For deep learning / neural networks, I think these are the best balance of theory and practice (not to mention great visualizations):
http://cs231n.github.io

More on the intuition/theory side for NNs, but Michael Nielsen’s book is highly recommended:
http://neuralnetworksanddeeplearning.com

For NLP, you can try this extremely slick interactive course on spaCy (highly recommended Swiss Army knife library for NLP) from package author Ines Montani:
https://course.spacy.io

The Keras tutorials are solid, here’s one on seq2seq models:
https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html

For more cutting edge NLP, it looks like the fast.ai course covers the Transformer model (the basis of BERT, GPT-2, et al. This is a great overview of that architecture:
http://nlp.seas.harvard.edu/2018/04/03/attention.html

And PyTorch implementations of Transformer models can be found here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/README.md

LMK if anyone wants more, I have a learning resource hoarding problem.

The universal approximator approach is another big one that people lean on from the opposite direction (I suppose you could call it the "anti-deep learning" school of thought). Most people don't take the time to consider what a universal approximator actually is.

Are humans universal approximators? Absolutely not. The mere proof that a nn with an infinite number of hidden nodes (which isn't practical, which could end the discussion there) can find a statistical mapping between any two input and outputs is hardly a benefit. There is something so much more to intelligence --- the ability to generalize well (aka not overfitting) is first and foremost.

I'd push you to ask why you perceive that stacking RBMs produces "eerily human-like" results. I'd argue that they don't: http://blog.keras.io/how-convolutional-neural-networks-see-the-world.html

In general the trend is toward more complex architectures, which increasingly blurs the line between modular nets and sparsely connected parts of a single model. IMO it's only meaningful to talk about modularity if the modules can be separately trained, or at least have their own loss functions. But of course its always useful to train or finetune the entire system end-to-end.

A semi-recent success story is Tacotron 2, a speech synthesizer that has one model to convert text to a spectrogram, and a second model to convert that spectrogram into pcm (waveform) data.

Also, Francois Chollet, the author of the popular deep learning library Keras, has some interesting thoughts on the future of ML along these lines that I find compelling: https://blog.keras.io/the-future-of-deep-learning.html