Name: Foundations of Statistical Natural Language Processing
ISBN: 9780262133609

Foundations of Statistical Natural Language Processing

Statistical approaches to processing natural language text have become dominant in recent years. This foundational text is the first comprehensive introduction to statistical natural language processing (NLP) to appear. The book contains all the theory and algorithms needed for building NLP tools. It provides broad but rigorous coverage of mathematical and linguistic foundations, as well as detailed discussion of statistical methods, allowing students and researchers to construct their own implementations. The book covers collocation finding, word sense disambiguation, probabilistic parsing, information retrieval, and other applications.

What Hacker News says

languagehackerDec 2025

Good old Jurafsky and Martin. Got to meet Dan Jurafsky when he visited UT back in '07 or so -- cool guy.

This one and Manning and Schutze's "Dice Book" (Foundations of Statistical Natural Language Processing) were what got me into computational linguistics, and eventually web development.

thread

YeGoblynQueenneOct 2023

>> GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model?

I believe GPT-4 was trained on a server farm, not a single computer. In any case what it was doing all that time was going over and over the text in its gigantic training corpus, which was of petabyte size, and optimising the objective P(tₖ|tₖ₋ₙ , ..., tₖ₋₁) i.e. the probability of token k given a "sliding window" of the n preceding (or surrounding) tokens.

There is nothing in this objective that needs a world model, and it is really not obvious why optimising this objective should lead to development of a world model, rather than, or in addition to, a model of the training corpus.

It is easy to see how this stuff works. You can train your own language model easily, although of course it would have to be a smaller language model. For example, you can train a Hidden Markov Model on the text of freely available literary works on project Guttenberg, or on wikipedia pages, and without too much compute (an ordinary laptop will do).

In fact, I recommend that as an exercise and as an experiment to gain a better understanding in how language modelling works, for those who are curious about questions regarding their ability to model something beyond text.

A good textbook to begin with statistical language modelling is "Foundations of Statistical Natural Language Processing" by Manning and Schűtze:

https://nlp.stanford.edu/fsnlp/

Or, just as good, "Speech and Language Processing" by Jurafsky and Martin:

https://web.stanford.edu/~jurafsky/slp3/

Or, if you don't have the time for an entire textbook, "Statistical Language Learning" by Eugene Charniak is an excellent, concise introduction to the subject:

https://archive.org/details/statisticallangu0000char

thread

YeGoblynQueenneMar 2023

Well, that's awkward. I didn't realise you were on HN. I'm sorry for the personal tone of my comment. You are right that it was uncalled for.

The paper you linked is clear on the scope of its proofs and in any case it's a very big assumption to say that "neural nets are Turing complete", when there are scant few such proofs, compared with the large number of different architectures (for most of which, no careful investigation of their computational capabilities is ever done anyway).

You could add a clarification to your article.

>> Do you have an alternative viewpoint on what allows LLMs to be able to somewhat accurately answer complicated math questions, despite lacking an explicitly programmed math solver?

Yes, it's because they're language models. In particular, they're very powerful, very smooth (in the statistical sense) language models trained to represent gigantic text corpora. Their ability to produce correct answers once in a while is not a surprise and does not need any other explanation.

Predicting what a language model (big or small) will output is another matter, so one particular instance of generated output might be surprising in the sense that the user won't expect it - not in the sense that the model shouldn't be able to produce it.

In any case, it's clear that the performance of those models depends on the prompts. Change the prompt slightly and you get a different answer, to any question. That suggests retrieval from memory (modulo stochasticity) much more than it suggests computation. And we know that these models are not models of computation, so there's no question what's really going on.

When I say "retrieval from memory" I don't mean that these models memorise whole sequences of tokens verbatim. To make a very big fudge about it, it's as if they've memorised templates that they can then apply to questions to generate the right answers.

I guess that still sounds magickal and mysterious if one hasn't worked with language models before, so all I can say is, if you are really curious, and really want to understand the specifics, you should try to learn more about language models.

I suggest the following as a starting point:

Eugene Charniak, Statistical Language Learning

https://mitpress.mit.edu/9780262531412/statistical-language-...

Dan Jurafsky and James H. Martin, Speech and Language Processing

https://web.stanford.edu/~jurafsky/slp3/

Chris Manning and Hinrich Schűtze, Foundations of Statistical Natural Language Processing

https://nlp.stanford.edu/fsnlp/

Those are rather "wax-on, wax-off", but if you want to learn Karate, that's where to begin. Then you can go on to beat up the Transformers and win the girl.

The Charniak book in particular is small and sweet and easy to read. Start there.

thread

YeGoblynQueenneApr 2019

Personally, I started with Eugene Charniak's Statistical Language Learning [1] then continued with Manning and Schütze's Foundations of Statistical Natural Language Processing [2] and Speech and Language Processing by Jurafsky and Martin [3].

The Charniak book is primarily about HMMs and quite short, so it's the best introduction to the subject. Manning and Schütze and Jurafsky and Martin are much more extensive and cover pretty much all of statistical NLP up to their publication date (so no LSTMs if I remember correctly) but they are required reading for an in-depth approach.

You will definitely want to go beyond HMMs at some point, so you will probably want the other two books. But, if you really just want to know about HMMs, then start with the Charniak.

______________

[1] https://mitpress.mit.edu/books/statistical-language-learning

[2] https://nlp.stanford.edu/fsnlp/

[3] https://web.stanford.edu/~jurafsky/slp3/

thread

eindiranOct 2018

For speech recognition, a few very foundational textbooks are:

* Foundations of Statistical Natural Language Processing by Manning and Schuetze [0]

* Statistical Methods for Speech Recognition by Jelinek [1]

* Spoken Language Processing: A Guide to Theory, Algorithm, and System Development by Huang, Acero and Hon [2]

In particular, Statistical Methods for Speech Recognition is a book you could expect to see on the shelves of most people in the field.

[0] https://www.amazon.com/Foundations-Statistical-Natural-Langu...