AI researcher and co-developer of LSTM networks, who writes on the history of deep learning, creativity theory, and credit for ideas.
Commenters find the site a rabbit hole worth exploring, note its distinctive design, and cite his claims that others were wrongly credited for deep learning work.
Ah yes, Perceptron. Had a couple typos.. sorry, was on phone.
The bio-inspiration was via Frank Rosenblatt, who is referred to in that article tho yeah, the history is over in his article:
https://en.wikipedia.org/wiki/Frank_Rosenblatt#Perceptron
"Rosenblatt was best known for the Perceptron, an electronic device which was constructed in accordance with biological principles and showed an ability to learn.
He developed and extended this approach in numerous papers and a book called Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, published by Spartan Books in 1962.[6] He received international recognition for the Perceptron.
The Mark I Perceptron, which is generally recognized as a forerunner to artificial intelligence, currently resides in the Smithsonian Institution in Washington D.C."
Your Juergen page is interesting, tho no direct comment on Rosenblatt there. He does cite the work on this page:
https://people.idsia.ch/~juergen/deep-learning-overview.html (refs R58, R61)
My reading is that a long-known idea, about multi-variate regression, was reinterpreted by Rosenblatt by 1958 via the bio-inspired Perceptron, and then that was criticized by Minksy and others and viable methods were achieved by 1965. When I was taught NNs by Mitchell at CMU in the 1990s (lectures similar to his book Machine Learning), this was the same basic story. Also reminds me of a moment in class one day when a Stats Prof who was surveying the course broke out…
Hinton didn’t invent back prop.
> Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971).
> BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980).
> To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers.
https://people.idsia.ch/~juergen/who-invented-backpropagation-2014.html
Hinton wasn’t the first to…
what it is they have against
repetition
They find it boring.
Would you come to HN every day if the linked articles never
changed?
I'm quite enamoured with Schmidthuber's theory of creativity [1, 2]
Schmidhuber develops a theory of
artificial curiosity and creativity for an autonomous agent. The
agent is equipped with an adaptive predictor trying to predict
future events from the history of previous events and actions. A
reward-maximising, reinforcement learning, adaptive controller is
steering the controller and gets curiosity reward for executing
action sequences that improve the predictor. (I term this "Pattern
Recognition as Pleasure".) This discourages the agent from
executing actions leading to boring outcomes that are either
predictable or totally unpredictable. Instead the controller
is motivated to learn actions that help the predictor to learn
new, previously unknown regularities in its environment, thus
improving its model of the world, which in turn can greatly help
to solve externally given tasks. Schmidhuber claims that his
corresponding formal theory of creativity explains essential
aspects of art, science, music, and humor.
If we were to take Schmidhuber's approach seriously, then that that which we enjoy as great…
I'm having trouble finding the source of this quote, but here:
>Many mathematicians have the opposite opinion; they do not or cannot distinguish the beauty or importance of a theorem from its proof. A theorem that is first published with a long and difficult proof is highly regarded. Someone who, preferably many years later, finds a short proof is "brilliant." But if the short proof had been obtained in the beginning, the theorem might have been disparaged as an "easy result." Erdős was a genius at finding brilliantly simple proofs of deep results, but, until recently, much of his work was ignored by the mathematical establishment.
And here is an interesting study on what mathematicians consider beautiful: https://www.gwern.net/docs/math/1990-wells.pdf They found that mathematicians varied a great deal in what they considered beautiful. And that there isn't any consensus on what is considered beautiful. It also changes over time:
>There was a notable number of low scores for the high
rank theorems 9 Le Lionnais has one explanation [7]:
"Euler's formula e^pi*i = - 1 establishes what appeared
in its time to be a fantastic connection between the
most important numbers in mathematics... It was
generally considered 'the most beautiful formula of
mathematics' . . . Today the intrinsic reason for this
compatibility has become so obvious that the same
formula now seems, if not insipid, at least entirely natural."
I think the interestingness or surprisingness of a mathematical…
Yes, Schmidhuber could present things in a better way and be more charitable. But do not overlook his work -- he has genuinely groundbreaking contributions to early theory of intelligence.
I recommend having a stroll through his website, for anyone interested in AI (or even art, creativity, etc.):
https://people.idsia.ch/~juergen/
Special Notes:
Godel Machine : While clearly absolutely impractical, this is a nice theoretical construction on what we could mean by "self-improvement" in a real, definite way.
Speed Priors
Super Omegas (and various thoughts on algorithmic information)
Always keep in mind a grain of salt when reading his results, but they are genuinely great.
---
I think establishing causation and giving credit does have a role in research, namely giving additional opportunity to those researchers to do even better research, elucidating the history of science, and to an extent giving a monetary incentive and morale recognition for researchers w.r.t. prizes, grants, etc.
For example isn't the story of Einstein, or von Neuman, or Feynman, etc. part of a narrative that makes science compelling? They had very interesting and lively lives and opinions that sometimes drove their research tastes. There is value in preserving individual history (as well as disregarding it when not particularly relevant).
That said, a scientists (or a good citizen in general) should not be driven primarily by ego, and in science this becomes clearly bad, impeding open…
So the unconscious mind is (metaphorically) like a powerful massively parallel computer and the conscious mind is like a shell or GUI.
Most of us are running "programs" that make us look, act, and feel like normal humans. (Anthropologists have pointed out that each culture, no matter how different from the others, manages to get its children to act like its adults.)
If you learn to reprogram your unconscious patterns you become a meta-computer: a computer that can rewrite its own software to improve itself, a Gödel machine[1].
The main reason why you don't "formulate well-thought responses in real time" is that your ad hoc programming doesn't include that functionality, but it's there, intrinsic to your capabilities as a human being.
Hypnosis is a way to put yourself into a sort of "debug mode" and communicate between your conscious and unconscious minds, and, if you like, reprogram your patterns. It's pretty easy to do.
In a nutshell, learn to go into trance[2], then tell yourself a story in which you discover how to "formulate well-thought responses in real time" quickly and easily. When you come out of trance, you will.
Feel free to ask questions, I'll do my best to answer. Most people aren't interested, so I appreciate the opportunity to pass along what I've learned. Cheers!
[1] https://people.idsia.ch/~juergen/goedelmachine.html
[2] I recommend "TRANCE-formations" by Bandler and Grinder.