Colin Raffel

Didn't say anything about algorithms improving faster than hardware. With better initialization and regularization, more efficient layers discovered by AutoML, sparsity, lower precision, or by directly combining models trained separately on diverse tasks instead of training sequentially we could have better models at the same hardware cost.

Even by improving the way people collaborate on model training we could make a huge leap. Instead of retraining from scratch we could reuse and update existing models. [1]

[1] A Call to Build Models Like We Build Open-Source Software https://colinraffel.com/blog/a-call-to-build-models-like-we-build-open-source-software.html

Thank you! I found a great list of deep learning paper summaries that is mostly about learning algorithms from data and other cutting edge models: http://colinraffel.com/wiki/tag:neural_networks?do=showtag&tag=Neural_Networks all kudos to the owner/author of this list !

>NNs that manipulate existing code? e.g. parse it, find design pattern, optimize

Also: you probably shouldn't need special architectures for such problems, these can be solved by seq2seq, NTM, RL-NTM and such, given a good dataset. Such tasks are still beyond current state of art though, it seems.

NES APU sound synthesis chip is an ideal constraint for cross-domain learning with Transformers ;)

LakhNES: Improving multi-instrumental music generation with cross-domain pre-training

https://chrisdonahue.com/LakhNES/

The NES Music Database (5k+ scores)

https://github.com/chrisdonahue/nesmdb

The Lakh MIDI Dataset v0.1 (175k+ files)

https://colinraffel.com/projects/lmd/