Filip Piekniewski writes skeptical posts on deep learning, self-driving cars and the prospect of an AI winter.
HN links it in discussions of AI hype and self-driving limits, and commenters engage with its points on prediction and the cost of closing the last gaps.
The process you are describing can be automated a bit.
Imagine your feed reader automatically subscribes you to feeds of articles you liked (star/thumbs up/favorite). Let's say you start subscribed to an HN feed of items that got more than 100 upvotes: https://hnrss.org/newest?points=100&count=100
Then you find the OP link and like it in your feed reader: https://www.niemanlab.org/2022/12/this-is-the-year-of-the-rss-reader-really/
Your reader automatically subscribes you to the source feed of this article: https://feeds.feedburner.com/NiemanJournalismLab
You like more content => you get subscribed to more feeds automatically.
Your feed reader can also take care of unsubscribing you from feeds that are posting too often or are not posting interesting to you content. If you don't like the content of a feed for awhile then your connection to that feed becomes weaker and the content from that feed is ranked less prominently. The content from feeds with a high signal-to-noise ratio (ie, how often you like content from that feed) is ranked higher for you.
This is how my little project https://linklonk.com works. If you'd like to give it a try: "thumb up" a few items from the above HN feed: https://linklonk.com/feed?url=https:%2F%2Fhnrss.org%2Fnewest%3Fpoints%3D100 and then check your recommendations on the home page.
Through this mechanism I am currently subscribed to 201 RSS feeds and I know that I won't miss content from high "signal-to-noise ratio" feeds. Here is a sample…
That reminds me of deep learning. Investment in computing resources is definitely increasing exponentially over time:
https://openai.com/blog/ai-and-compute/
https://techcrunch.com/2019/11/10/the-post-exponential-era-of-ai-and-moores-law/
Tthere doesn't seem to by any evidence of exponential gains in performance, even on their own benchmarks (which importantly don't measure "intelligence", but something more narrow):
https://blog.piekniewski.info/2019/11/18/late2019-the-wizards-of-oz/
Even though we are doubling the compute available to AI every 3.5 months, so a factor 10x per year, since say 2015, so some 10.000x, the classification performance as measured by top-1 accuracy on ImageNet had barely moved. Now arguably top-1 accuracy on ImageNet is probably not the best measure, but still that seems somewhat striking. The diminishing returns are visible elsewhere too, even that Open AI Rubic's cube is somewhat indicative.
From an outside perspective, it seems that everyone is coming around to the fact that self-driving cars is harder than expected. There are no "exponential gains" in that field -- to the contrary, it seems like 90% of the problem was solved several years ago, and you can now take undergraduate courses that teach the basics.
But going from 90%, to 99%, to 99.9%, etc. is taking decades, not months or years. There seems to be something fundamentally wrong with the current approach.
I believe at the time the OpenAI blog post was supposed to be…
There are dozens of papers "explaining" neural networks, using basic math, highly abstract math and even theoretical quantum physics[1] models. I don't see any of them used in constructing, improving or interpreting results of actual neural networks[0]. Moreover, while most of them have math that would take me weeks to fully understand, they rarely make concrete, testable predictions.
Most of the recent advances in ANN learning came from using more data and using different types of layers. A real theory of neural nets would explain why architecture A works better than architecture B for a particular problem. Or why a certain architecture plateaus after working with X samples, while other can learn from many more.
The way I understand it, the core problem of understanding deep learning is that it does not scale down. You can make a simple framework of 3-4 neurons and observe how it works. Then you make it bigger and it works differently. No one properly explained why this happens. People who do applications seem to just state their intuitions and then throw a couple formulas on top to make it seem scientific. There is a great essay on that by Piekniewski[2]. No research I've seen lists all the parameters/architectures/datasets they tried that didn't work.
[0] - Maybe with the exception of the paper on neural networks as ODEs.
[1] - http://neurophysics.huji.ac.il/sites/default/files/PhysRevA.32.1007.pdf
[2] -…
Yes,
The thing about self-driving you can get to 90% or 99% or even 99.9% success and then find the cost of closing part of that extra sliver of non-success is the same as all the work you've done. Especially since testing those corners cases is hard since they are unusual. There's even some part of the sliver that even is "AI complete". Very small but you don't know where it is, etc.
But I'd say that's going to happen about as quickly as changing the infrastructure in this country.
Key point!
Automation has traditionally been much, much easier when you are engineering a situation where you control all the variables. Automated factories are common. Automated warehouses are coming. Automated dock systems exist. In these situation, there's a hard border between "inside" and "outside". Roads are generally regular but the border between insight and outside fuzzy. Google can run it's self-driving taxis in a suburb in Arizona where roads are as regular as could be imagined but expanding beyond that is much, much hard than one imagines.
Note: I recommend this blog which periodically tracks progress in self-driving cars as well as making intelligent, critical comments on current AI.
https://blog.piekniewski.info/
I don't know why HN loves Sam Altman so much. Has he ever written a significant piece of software or invented anything? Has he ever built a significant, profitable business from the ground up? The guy was a fundraiser superstar when money was easy to get in the 2010s but I have yet to see technical or business competence sufficient to grant him the level of respect many in this community seem to grant him.
There are people who are technically competent or genuinely intelligent who can also raise funds.
The guy reminds me of Elizabeth Holmes or Eliezer Yudkowski of "lesswrong."
A sophomoric guy with minimal technical or business accomplishments who is just smart enough to tell rich people what they wanna hear to fuel his game.
There is an old term for this personality type. Snake oil salesman aka con artist.
See https://blog.piekniewski.info/
For a sane take on AI by someone who actually got an education and is an expert in his field.
In a sense, I would say yes there are learning laws, but it's still early in codifying them.
Along one axis, you could compare: supervised, semi-supervised, self-supervised and unsupervised learning. Along another axis, consider that there are versions of each method that take into account temporal/dynamic data, versus others that require randomly shuffled static data.
In the current problems of visual perception, I think the field would benefit greatly a shift to focus on multiscale interaction/dynamics rather than on (static) statistics as it is currently (for more on this, see my colleague's blog: [1]).
[1] Statistics and dynamics. (http://blog.piekniewski.info/2016/11/01/statistics-and-dynamics/)