Another slick trick is that latex (really, tex) lets you alter the weightings used in the optimization, paragraph by paragraph. And you can alter the weightings used in deciding where to put page breaks. You can also supply hints on hyphenation, for those rare cases in which the engine cannot decide what to do. (Note: the system adjusts hyphenation to the language, so e.g. those long words in German will be handled well.)
The optimization scheme is a large part of what makes tex so good. That, and the understanding of mathematical notation.
Anyone who wants to get insights into the early setup of tex might enjoy reading the following Knuth (1979). It's a very engaging read, for something that's technical. (Generally, Knuth is a great writer.)
References
Knuth, Donald E. “Mathematical Typography.” Bull. Amer. Math. Soc. (N.S.) 1, no. 2 (March 1979): 337–72.
The motivation was not to replace art with technology, but to preserve/resurrect an art that was going away, by truly capturing the human understanding[^1]. When the rest of the industry was perfectly content with the deterioration of typesetting, Knuth set out to capture the aesthetics of the best journals of the past. A quote from the Mathematical Typography paper I linked above (https://websites.umich.edu/~millerpd//docs/501_Winter13/Knut...):
> At this point I regretfully stopped submitting papers to the American Mathematical Society, since the finished product was just too painful for me to look at. Similar fluctuations of typographical quality have appeared recently in all technical fields, especially in physics where the situation has gotten even worse.
Frankly, I think the "replace art by technology" impression is a very shallow one, that I alluded to earlier. When Knuth wrote his “The Concept of a Meta-Font” in a journal (Visible Language) mostly read by designers/typographers, many of them wrote letters in response (https://shreevatsa.net/tex/metafont/concept#reactions). What you can see is that the best of them were supportive (even bringing up new points like how it could be useful in educating the next generation of font designers), but some were sharply critical, more or less resenting this intrusion of technology into their art medium. But now a few decades later, basically all fonts are distributed and stored digitally anyway, except that (without METAFONT) the shapes of letters are now basically just stored as binary blobs / sequences of numbers, without any METAFONT-like understanding of typographically relevant quantities like (say) x-height, comma depth, slab thickness, etc. Which one is truer to the art?
(Not a rhetorical question BTW: as in the Bigelow/Southall quote above, one could say that Knuth's approach is to achieve typographical/artistic excellence through understanding, but the artistic approach is visual and intuitive without a cognitive component. But this is a different complaint from the "replace art by technology" take.)
(BTW apart from the default Computer Modern fonts designed by Knuth, who based them on earlier Monotype fonts, almost all fonts used by people with TeX too are designed by font designers, not computer scientists.)
[1]: Related quote from Knuth (sorry paraphrasing from memory): “People say that the best way to understand something is to teach it. I say no: the best way to understand something is to teach it to a computer.” But then again he has also said: "Science is what we understand well enough to explain to a computer. Art is everything else we do."
Maybe I'm such a fan because of many great experiences with tex (and latex). The output is just so beautiful. And the notation (once you get used to the backslashes etc) is so clean.
But, heck, I got hooked when I studied Figure 12 of [1], as an undergraduate leafing through a math journal. At first, I thought "what's wrong with this nutcase, those "s" letters all look fine to me". Then I started to see differences. And then some looked ugly. That one in the middle, though, looked better and better, the more I studied the figure. Pretty soon, I was hooked.
1. Knuth, Donald E. “Mathematical Typography.” Bull. Amer. Math. Soc. (N.S.) 1, no. 2 (March 1979): 337–72.
• During 1977–78, Knuth wrote the first version of TeX (now called TeX78) in SAIL and gave a talk[1] about it. Everyone was excited, and lots of people at other universities/labs starting writing their own ports of TeX.
• During 1980–82, Knuth set out to rewrite a “portable” TeX using Pascal, the most widely available language at that time (actually he wrote his own macros/wrapper on top of Pascal; the resulting language is called WEB). This (TeX82 aka TeX) is the program still in use today if you invoke "tex". (Continues to be updated once every few years, but basically stable.)
• During 1998–2000, there was a Java rewrite of TeX called the New Typesetting System (NTS)[2], which had an alpha release and then was basically abandoned: it had all the problems of TeX and was several times slower, so it was regarded by many as a failure. Meanwhile extensions of TeX like pdfTeX came into existence, which are the programs most people use today.
• The code still exists though, and I imagine people must have tried it on and off. But since I can't find any earlier record, I'll provisionally take credit for the idea of resurrecting/resurfacing it, when I tried it in 2017: [3]
• Recently (in the last few months), Dave Jarvis (user `thangalin` here) has forked/modernized this Java implementation and is trying it out. (I noticed when he left his comment on my TeX.SE question; I guess his post https://tex.stackexchange.com/a/672614 on another question is what led to it being posted here.) I'm excited about this; looking forward to see how it gets used.
(Apologies if this long comment potentially takes attention away from the current state and future possibilities, but felt like the history is useful context…)
————
I'll also add some history about the license, as I see another thread about it. When Knuth wrote TeX, it was to solve a specific problem he had encountered in publishing: the second edition of TAOCP Vol 2 was turning out worse in appearance than the first, because the publisher was moving from hot-metal typesetting (with Monotype machines) to phototypesetting, and could not achieve exactly the same appearance. While being bothered by this, Knuth discovered that digital typesetting machines were starting to come into existence: now it was just 0s and 1s, and as a programmer he felt he could handle it. That was his motivation for creating TeX and METAFONT: the appearance of each page would remain reproducible forever into the future.
Except: when he publicized TeX and various clones sprung up, the situation was heading towards merely re-introducing the problem: if different computer systems had different programs all called “tex” that made even subtly different choices about line-breaking or whatever, then someone could write a paper/book on one computer, fine-tune the typesetting, then send the .tex file to their colleague / editor / publisher who used another computer, and no longer be confident that they would get the same result. This was one of the reasons he used the rewrite to create a portable TeX, going to great pains to use only a common subset of Pascal (Pascal had widely diverging compiler and language features), wrote a very demanding conformance test suite called the TRIP test[4] and insisted that any program would need to pass this test to be called "tex" — that way, everyone using "tex" could be sure they were writing something that would always produce the same result everywhere. If you wanted to make different choices in the program you could, as long as you changed the name.
When he announced in 1990[5] that TeX would be stable (no more features), this is what he wrote:
> My work on developing TeX, METAFONT, and Computer Modern has come to an end. I will make no further changes except to correct extremely serious bugs. I have put these systems into the public domain so that people everywhere can use the ideas freely if they wish. I have also spent thousands of hours trying to ensure that the systems produce essentially identical results on all computers. […]
> […] anybody can make use of my programs in whatever way they wish, as long as they do not use the names TeX, METAFONT, or Computer Modern. In particular, any person or group who wants to produce a program superior to mine is free to do so. However, nobody is allowed to call a system TeX or METAFONT unless that system conforms 100% to my own programs […]
> I welcome continued research that will lead to alternative systems that can typeset documents better than TeX is able to do. But the authors of such systems must think of another name. That is all I ask, after devoting a substantial portion of my life to the creation of these systems and making them available to everybody in the world. I sincerely hope that the members of TUG will help me to enforce these wishes…
etc. And NTS derives a similar licence, and this project (being forked from NTS) simply carries over the license.
Of course such a license is unusual today, and thousands of pages' worth of ink have been spilled over whether or not this counts as “free software” (predating GPL / MIT license etc), and (finally) grudgingly accepted to be. You can read about the history of the LaTeX project's similar license (LPPL) and all the debates on Debian/FSF mailing lists etc if you're morbidly interested[6]; the discussion usually proceeds on similar lines and I hope it doesn't get repeated here. :-)
[1]: Gibbs lecture, “Mathematical Typography”: https://doi.org/10.1090/S0273-0979-1979-14598-1
[2]: https://en.wikipedia.org/w/index.php?title=New_Typesetting_S...
[3]: https://tex.stackexchange.com/questions/385645/is-nts-new-ty...
[4]: http://mirrors.ctan.org/info/knuth-pdf/tex/tripman.pdf
