Neural networks for pattern recognition

What Hacker News says

1. plinkplonkOct 2010
   "Learning AI (or any deep comp.sci for that matter) is not like learning J2EE or Ruby DSLs or whatever the fad du jour in the enterprise software world is — read a few chapters of the latest bestselling “pragmatic” book, write some crappy web site and hey presto you are the expert.

   This seems like over-the-top smuggery."

   That's probably because he didn't say it.

   I did (I don't know if you noticed he was linking to a separate page, and if so you followed the link), a long time ago, in a blog post. And of course the context was different.

   Here is the original post from which that sentence was extracted. Hopefully it makes more sense in context.

   -------------------------------------------------------

   "To "learn AI" <---- title of the blog post.

   first learn basic mathematics.

   I keep getting asked this question and I keep saying the same thing - to three people in the last week, for e.g, two of whom were working through (or planning to work through) AIMA - so I thought I'd put this down here (and point anyone who asks the same question to this entry in the future).

   Learning AI (or any deep comp.sci for that matter) is not like learning J2EE or ruby "dsl"s or whatever the fad du jour in the enterprise software world is - read a few chapters of the latest bestselling "pragmatic" book, write some crappy web site and hey presto you are the expert.

   "Real" comp sci doesn't quite work like that.

   To really understand a standard 3 layer feed forward neural network, for example, you need to have a solid grip on

       * vector spaces
       * basis vectors and change of bases
       * eigen vectors and eigen values
       * Basic matrix operations like inversion
       * multi dimensional performance surfaces
       * Quadratic functions and finding their global maxima within Newton's method and Conjugate gradients
       * performance measures and steepest descent
       * partial differentiation of vector values
       * numerical stability of algorithms
   

   Without that background you will be able to use conceive of "class NeuralNetwork" that "has an" instance of "class Node" etc, but you will not,(repeat NOT) be able to do anything useful with real world data (writing a XOR classifier doesn't count!).

   And, a feed forward neural network is only one type of pattern recognizer (or function approximator). There are many more, each with its own trade offs , and you have to know the math to make the trade offs.

   That being said, the best book to start with is AIMA as long as you (1) learn the required math parallely (2) do all the exercises at the end of every chapter.(I can't emphasize this enough). Reading through != "working through".

   If one is willing to work hard, there are very few fields as fascinating as the various branches of AI.

   Once you start down the rabbit hole, however it may be very hard to continue writing all that heavy lifting enterprise software without hearing the giant sucking sound of your life going down the drain as you write yet another jsp page for the latest leasing system. ;-)

   Be warned! :-D.

   Anyway the really hard part of "learning AI" is not getting the books or working through them systematically (which is hard enough), but that is for another blog entry.

   Post Script - Some book recommendations for beginners.

      1. Linear Algebra by Gilbert Strang
      2. Calculus by Gilbert Strang
      3. Artificial Intelligence A Modern Approach By Russell and Norvig
      4. Machine Learning By Tom Mitchell
      5. Neural Networks for Pattern Recognition by Chris Bishop
      6. Paradigms of Artificial Intelligence Programming by Peter Norvig (ok, this one is not quite an cutting edge AI book, but imo this is the best book on programming ever written and if you call yourself a programmer you should have it on your book shelf)
   

   end blog post.

   _______________________________________________________

   Hopefully that wasn't too much "over the top smuggery" :-). (Not that I care if it is, I write stuff on my blog to clear my head, not be popular).

   This was written as answer to my ex-colleagues from the world of enterprise software who asked for advice on "How to move into AI Dev and do interesting projects like you do" and the "leasing system " and "heavy lifting" "read some book and become the (local) expert" etc references make sense to the people (from ThoughtWorks) who asked me the question.

   I am not sure how anything I said fits into this Java-bank-interview vs Haskell-bank-interview discussion.

   Fwiw I am entirely self taught and was addressing other self learners in that blog post, so I am not sure how the author uses my opinion to buttress his thesis that "self learning is overrated". If anything I believe it is underrated.

   jrockway said it best above,

   " I've learned everything I know about programming on my own -- reading other people's code, and trying their techniques for myself. "

   Amen.

   Show morethread
2. plinkplonkJan 2010
   I wrote a supplementary blog post to Brad's ( http://pindancing.blogspot.com/2010/01/learning-about-machin...) with the list of books I found useful (no amazon referral links if anyone is worried) with brief descriptions of each. I work in somewhat different domains and so have a different list of books.

   I hope someone finds this useful.

   For those who want a summary,

   Proof Technique

   (a)Velleman's "How to Prove It" (b)Gries and Schneider's "A Logical Approach to Discrete Math"

   Math

   (c) Calculus (best "lite" book - Calculus by Strang (free download), best "heavy" books - (d) Calculus by Spivak, (e) Principles of Mathematical Analysis a.k.a "Baby Rudin")

   (f) Discrete Math (ALADM above + (g) a good book on Algorithms, Cormen will do - though working through it comprehensively is ... hard!

   (h) Linear Algebra (First work through Strang's book, then (i) Axler's)

   (j) Probability (see Bradford's very comprehensive recommendations) and

   (k) Statistics (I would reccomend Devore and Peck for the total beginner but it is a damn expensive book. So hit a library or get a bootlegged copy to see if it suits you before buying a copy, see brad's list for advanced stuff.)

   (l) Information Theory (MacKay's book is freely available online)

   Basic AI

   (m)AIMA 3d Edition (I prefer this to Mitchell)

   Machine Learning

   (n) "Pattern Recognition and Machine Learning" by Christopher Bishop,

   (o)"Elements of Statistical Learning" (free download).

   (p) Neural Network Design by Hagan Demuth and Kneale,

   (q) Neural Networks, A Comprehensive Foundation (2nd edition) - By Haykin (there is a newer edition out but I don't know anything about that, this is the one I used)

   (r) Neural Networks for Pattern Recognition ( Bishop).

   At this point you are in good shape to read any papers in NN. My reccomendations - anything by Yann LeCun and Geoffrey Hinton. Both do amazing research.

   Reinforcement Learning

   (s) Reinforcement Learning - An Introduction by Barto and Sutton (follow up with "Recent Advances In reinforcement Learning" (PDF) which is an old paper but a GREAT introduction to Hierarchical Reinforcement learning)

   (t) Neuro Dynamic Programming by Bertsekas

   Computer Vision

   (u) Introductory Techniques for 3-D Computer Vision, by Emanuele Trucco and Alessandro Verri.

   (v) An Invitation to 3-D Vision by Y. Ma, S. Soatto, J. Kosecka, S.S. Sastry. (warning TOUGH!!)

   Robotics.

   (w) Probabilistic Graphical Models: Principles and Techniques (Adaptive Computation and Machine Learning) - not about robotics per se but useful to understand the next book

   (x) Probabilistic Robotics (Intelligent Robotics and Autonomous Agents) by Thrun, Burgard and Fox

   PS: I own all these books (except AIMA 3 for which I only have pre pub pdfs) and if any HN folks in Bangalore want to browse before they buy anything (friggin expensive when you add amazon's postage) send me email.

   PPS: on languages, I think Bradford is on the money with regards to reccomending functional languages. I would just say, also know C well. Saved my ass a few times.

   Show morethread
3. signa11Jan 2009
   my recommendations (for neural networks) would be as follows:

      - parallel distributed processing (PDP group vol 1 & 2)
      - simon haykin (Neural Networks and Learning Machines)
      - christopher bishop (Neural Networks for Pattern Recognition)

   thread
4. brentJun 2008
   I agree about #2. I have Russell and Norvig and agree that it is an excellent book, but I am not sure how much overlap there truly is here.

   I also do not have PRML, but Neural Networks for Pattern Recognition by Bishop is excellent (and includes many non-NN related items).

   ESL is excellent and, to me, the best modern text in machine learning. It covers many of the topics in PCI both at a reasonable level and in much more depth and provides MANY references (100s?) to dig deeper.

   Show morethread
5. signa11Mar 2018
   indeed ! the 'neural networks for pattern recognition' book (bishop) has exercises in chapter-01 on the same subject.
   thread
6. oergiRDec 2016
   The "model" in the title is the model of the world, as a probabilistic model. The good thing about such a model is that it explicitly states your beliefs about the world. Once you've defined it, in theory reasoning about it is straightforward. (In practice a lot of papers get written about how to do approximate inference.) It's also straightforward to do unsupervised learning.

   This is a different perspective from (most uses of) neural networks, which do not have this clear separation between the model and how to reason about it. It's funny that Chris Bishop in 1995 wrote the textbook "Neural Networks for Pattern Recognition" and now is effectively arguing against using neural networks.

   You can use both by using neural networks as "factors" (the black squares) in probabilistic models.

   Show morethread