Carlos E. Perez (Intuition Machine)

What this particular model shows is probably about as simple as you can get; the activation function (what is called "predict" in the code) is a simple linear model (very similar to RELU - which has other important properties valuable for multiple layer models vs other activation functions). You probably couldn't make it more simple unless you made it a binary function (ie - set it to "1" if the value is anything over zero, otherwise set it to "0").

Everything else basically stays the same, though, and it can't get much simpler.

Here's the thing though: As far as anyone knows - and research is ongoing on this, so this may change at any time - heck, it may be different now than the last time I looked into it - but anyhow, as far as anyone knows, in nature (that is, actual neurons in a brain), backpropagation is not done, nor is there an apparent mechanism to allow for it.

This has led Geoffrey Hinton to think maybe things should be rethought (and he has gone down the road of something called "capsule networks" which I have yet to understand - I can't tell if they are actually simpler or more complex from what I have read):

https://medium.com/intuitionmachine/the-deeply-suspicious-nature-of-backpropagation-9bed5e2b085e

This doesn't mean that backpropagation doesn't exist in nature, just that we haven't found a mechanism for it (although there are some competing theories).

I have also read research papers on other ways of making a neural network learn - one was something…

I think they're on to something. Our constraint on abstraction layers that they must be legible and grokkable by human minds, that's just something we impose because we feel that we should understand. But that constraint is overburden. I feel that neural nets are starting to prod at this. Our biology works and does incredibly complex computation (including social computation), and it's creation process didn't give a fuck whether a human could understand it. We're still playing catch-up with that sort of low-power computation that biology does. A sandwich powers a wildly powerful neural network

This also reminds me of how alien and organic ML-assisted design is -- when a design task is constrained only by its environmental constraints, and human aesthetic and comforts are discarded: http://jcsitnet.com/journals/jcsit/Vol_7_No_2_December_2019/7.pdf https://medium.com/intuitionmachine/the-alien-look-of-deep-learning-generative-design-5c5f871f7d10

Straight lines and clean edges and nice angles are just abstractions that we humans happen to have preference for due to our very particular evolutionary history. As in, we happen to like clean straight lines and certain logics in physical and visual domain, but have no such constraints on taste and smell preferences, and in fact love complexity in those sensory domains, because that served us in our past ecological niches. But discarding these visual/logical sensibilities and allowing abstraction interfaces to float free of our…

Ahah, good point. Really the ETH stuff was "because I can". But in the same charts repository you will find a Tensorflow chart. My previous series of blogs [0] was about exactly that. A nice addition as well for compute intensive workloads is the use of LXD [1]

Another use case is in media for transcoding. It is not a trivial job to orchestrate transcoding at scale, and Kubernetes with or without GPUs is an excellent solution for that as it is trivial to setup a completely automated job queue.

Also another interesting field will eventually be HPC but there are some constraints about compute that K8s does not tick scheduling wise at this point in time. There is a pluggable scheduler in the works I think, and this will eventually help. Also the LXD example is a nice optimization but it would not replace the scheduler in any way.

[0]: https://medium.com/intuitionmachine/gpus-kubernetes-for-deep-learning-part-3-3-automating-tensorflow-deployment-5dc4d5472e91

[1]: https://hackernoon.com/job-concurrency-in-kubernetes-lxd-and-cpu-pinning-to-the-rescue-b9fb7b44f99d

Of course. As others here point out, the hypervolume inside an n-dimensional hypersphere grows as the nth power of a linear increase in radius. In high dimensions, tiny increases in radius cause hypervolume to grow by more than 100%. The concentration of hypervolume is always highest at the edge.[0]

The theoretical tools (and intuitions) we have today for making sense of the distribution of data, developed over the past three centuries, break down in high dimensions. The fact that in high dimensions Gaussian distributions are not "clouds" but actually "soap bubbles" is a perfect example of this breakdown. Can you imagine trying to model a cloud of high-dimensional points lying on or near a lower-dimensional manifold with soap bubbles?

If the data is not only high-dimensional but also non-linearly entangled, we don't yet have "mental tools" for reasoning about it:

* https://medium.com/intuitionmachine/why-probability-theory-should-be-thrown-under-the-bus-36e5d69a34c9

* https://news.ycombinator.com/item?id=15620794

[0] See kgwgk's comment below.

I think if you could somehow start computing the resultant matrix elements as soon as you read a row/column from the input ones, you could reach their "physically possible" limit.

A couch expert on computer architecture here, but a large enough systolic array could be used to achieve their "physically possible" limit? [0]

New CUDA GPUs have been coming with these tensor cores that are just systolic arrays. Google's TPU are the same.

Could someone with more knowledge on systolic arrays comment on whether a large systolic array can achieve this?

[0] https://medium.com/intuitionmachine/googles-ai-processor-is-inspired-by-the-heart-d0f01b72defe