Eric Jang

The superformula depends of four parameters and is able to model many different curves. I wonder if that superformula would be useful to learn to generalize the form of a curve given few points. It could be that, in same way, the four parameters of that curve are a orthogonal bases in the hypothesis space, in the sense that each parameters add a lot the information. If this intuition has any meaning, it could be the start of a new theory for constructing bases of the hypothesis space, that is models with few parameters but great expressive power.

Edited: (1) The following link explains expressivity and generalization power in machine learning: https://blog.evjang.com/2017/11/exp-train-gen.html

So my question is whether the superformula constitute an example of great expressivity and powerful generalization for curve fitting by using machine learning models.

Edited: (2) In the following link they use the superformula, Automatic Generation of Smooth Curves from Interpretable Low-Dimensional Parameters.

So the intuition seems fruitful. https://arxiv.org/pdf/1808.08871.pdf

I appreciate the authors calling out #2. ImageNet and CIFAR are both "solved" benchmarks, to the extent that state of the art algorithms by now are likely overfitting to the specific dataset details. In particular, objects in ImageNet are nearly always in canonical pose, with no occlusion, few confounding objects, illuminated, and in a semantically-obvious configuration. For some applications this is acceptable but as an industry benchmark ImageNet is not informative (user generated photo distributions are never this clean).

Even more damning is the recent BagNet paper (nice summary here: https://blog.evjang.com/2019/02/bagnet.html), which indicates that ImageNet can likely be solved with no global features (i.e. model doesn't have to learn anything truly abstract, just configurations of textures, shapes, colors). I thought the author of that blog post put it nicely:

"As someone who is deeply interested in AGI, I find ImageNet much less interesting now, precisely because it can be solved with models that have little global understanding of images."

I agree that the sort of sampling bias/distributional shift you mention is definitely a real problem in deep learning. It so happens that yesterday I read a blog post that elaborates on this problem clearer than I can. Although it's still true that allowing arbitrary distributional shift is impossible, it does make sense to allow for distributional shift across the natural manifold (manifold of all "allowable" data). The blog post called this "strong generalization", which isn't a term I've heard before.

This is the blog post I'm referring to: http://blog.evjang.com/2017/11/exp-train-gen.html?spref=tw

I'm just an undergraduate doing research in computer vision/deep learning. I've been following a lot of the research in deep learning theory because I think it's really interesting and there's been a lot of activity recently. Maybe I'll be able to do research on pure ML theory one day, but my math ability is definitely not up to par yet haha...

Right.I was focusing more on the CUDA-kernels-on-NPU line of thought from patrikthebold's message and its alternatives, which as you say, is not a reality now either.

In the Jax style implementation scenario the compiler part of JAX is better inspiration, maybe along the lines of this case study of a path tracer running on a TPU: https://blog.evjang.com/2019/11/jaxpt.html - I don't think Chapel or Futhark would adopt the same approach as such but it's at least some kind of existence proof of a compiler targeting it from a high level language for a non-machine learning code.

Ensoul | Member of Technical Staff | San Francisco, CA | ONSITE | $180,000-$300,000 + Equity + Benefits
https://ensoul.inc/t/099a40

Ensoul's mission is to accelerate robotics research. We're a team of frontier-lab scientists and roboticists working to accelerate robotics research and unlock The Great Robotics Buildout. Our customer is the robotics researcher.

The company page is light on details, so here's a bit more about me: https://evjang.com, I previously led AI at 1X Technologies and co-led some of the efforts at Google Robotics that led to SayCan, RT-1, etc.