Synced Review is a Medium publication that reports on AI research papers and industry news, including model training costs and new architectures.
HN links it for coverage of AI research results, such as Google's Fourier transform replacement for BERT attention and the cost of training large models.
- "Some of the models are so big that even in MILA we can’t run them because we don’t have the infrastructure for that. Only a few companies can run these very big models they’re talking about" [1]. NOTE: MILA is a very good AI research center and, while I don't know too much about him, that person being quoted has great credentials so I would generally trust them.
- "the current version of OpenAI Five has consumed 800 petaflop/s-days" [2].
- Check out the Green AI paper. They have good number on the amount of compute to train a model and you can translate that into numbers.
- https://medium.com/syncedreview/the-staggering-cost-of-training-sota-ai-models-e329e80fa82. NOTE: That XLNet number has to be wrong - it should be 5-figures, not 6.
I'm not an expert in on-prem ML costs, but I know many of the world's best on-prem ML users use the cloud to handle the variability of their workloads so I don't think on-prem is a magic bullet cost wise.
$1M annually per project (vs per lab) isn't bad at all. It's also way out of whack with what I saw when I was doing AI research in academia, but that was pre deep learning revolution, so what do I know.
Re: the moving goalposts - the distinction is between the cost of a training run and the cost of a paper-worth research result. Due to inherent variability, architecture search, hyperparameter search and possibly data cleaning work, the total cost is a couple orders of magnitude more than the cost of a training run (multiple will vary…
Wave Field recordings are probably the most complete known descriptions of the brain and its nonlinear fields?
How such fields relate to one or more Quantum Wave functions might entail near-necessity of QFT: Quantum Fourier Transform.
When you replace the Self-attention Network part of a Transformer algorithm with classical FFT Fast Fourier Transform: ... From https://medium.com/syncedreview/google-replaces-bert-self-attention-with-fourier-transform-92-accuracy-7-times-faster-on-gpus-7a78e3e4ac0e :
> > New research from a Google team proposes replacing the self-attention sublayers with simple linear transformations that “mix” input tokens to significantly speed up the transformer encoder with limited accuracy cost. Even more surprisingly, the team discovers that replacing the self-attention sublayer with a standard, unparameterized Fourier Transform achieves 92 percent of the accuracy of BERT on the GLUE benchmark, with training times that are seven times faster on GPUs and twice as fast on TPUs."
> > Would Transformers (with self-attention) make what things better? Maybe QFT? There are quantum chemical interactions in the brain. Are they necessary or relevant for what fidelity of emulation of a non-discrete brain?
> Quantum Fourier Transform: https://en.wikipedia.org/wiki/Quantum_Fourier_transform
> that's all ML seems to be.
You have to be very selective about what you consider "ML" to come to that conclusion. There has been a constant parade of incredible, mind-blowing results out of ML over the past decade, advancing the state of the art by leaps and bounds both in research and in real applications.
Do you not remember how terrible speech recognition and speech synthesis were just a few short years ago? Did you not see DeepMind finally crack Go? Check out BigGAN [1]. Try search in Google Photos. See translation getting better every year.
Yes, there are quacks and charlatans and people who are just plain wrong. But ML is real, and it solves real problems that people failed to solve any other way despite decades of concerted effort.
[1] https://medium.com/syncedreview/biggan-a-new-state-of-the-art-in-image-synthesis-cf2ec5694024
(I work at OpenAI.)
It comes down to whether you believe AGI is achievable.
We've talked about why we think it might be: https://medium.com/syncedreview/openai-founder-short-term-agi-is-a-serious-possibility-368424f7462f, https://www.youtube.com/watch?v=YHCSNsLKHfM
And we certainly have more of a plan for building it than warp drives :).
EDIT: I personally think the case for near-term AGI is strong enough that it'd be hard for me work on any other problem — and find it important to put in place guardrails like https://openai.com/blog/openai-lp/ and https://openai.com/charter/.
Even if AGI turns out to be out of reach, we'll still be creating increasingly powerful AI technologies — which I think pretty clearly have the potential to alter society and require special care and thought.
> It would be a shame if ML also became a field people avoided because they didn't want to contribute to evil, in their own view.
Both ethics and privacy considerations have recently become pretty regular at Computer Vision and Multimedia Processing conferences.
One very popular object detection model (called YOLO) had the main author recently leaving the field because he got concerned about military applications using his research results.
https://medium.com/syncedreview/yolo-creator-says-he-stopped-cv-research-due-to-ethical-concerns-b55a291ebb29
Background: https://seekingalpha.com/article/4206602-first-look-liulishuos-72-million-ipo
> The company has 83.8 million registered users, of which 1 million are paid users.
> The company's rapid growth (~6x in the first half of this year)
Per the F-1, the average Liulishuo user spends over 60 minutes per day learning English in the app.
How their product works: https://medium.com/syncedreview/liulishuos-ai-app-is-teaching-english-to-70-million-people-31d4fb38a799