The Hugging Face team writes about NLP models, benchmarks, distillation and training techniques for PyTorch and TensorFlow.
HN commenters link it for DistilBERT, the PyTorch versus TensorFlow benchmarks and its articles on sparse neural networks.
The memory constraint is certainly an issue, but I think they can be overcome with some software hacks (for a speed penalty, of course.) For example, gradient checkpointing and gradient accumulation might help:
https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9
https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255
> Edit: I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model.
Huggingface did just that:
https://medium.com/huggingface/distilbert-8cf3380435b5
OpenAI just announced it will port its sparse libraries to PyTorch, so exciting times ahead!
You can read more here about this (OP here) :
https://medium.com/huggingface/is-the-future-of-neural-networks-sparse-an-introduction-1-n-d03923ecbd70
It is from my own experience. But it looks like inference is a bit faster on CPU: https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2
This is similar to DistilBert and DistilGPT2 by HuggingFace
https://medium.com/huggingface/distilbert-8cf3380435b5