Name: Stochastic optimal control
ISBN: 9780120932603

What Hacker News says

graycatDec 2018

Calculus? Freshman college calculus, never took it. Just read a book and started the class on sophomore calculus.

Teaching calculus? Been there; done that, as a math grad student. Easy to teach -- off the back of my hand. Zero time for preparation.

Now with D. Knuth's TeX, it would be easy to pass out very nicely polished class notes, examples, exercises, applications, etc.

Here's an application that might interest the students. At one time it literally saved FedEx from going out of business.

The BoD wanted some revenue projections for the full, planned fleet and business. So, we knew what the revenue was then and what the revenue should be for the full planned fleet. So, the projections were essentially an interpolation.

So, there would be growth, and what would drive that growth? Okay, assume the rate of growth is proportional to (1) the number of current customers talking about FedEx and (2) the number of the rest of the target customers hearing the talking.

So, for time t, with t = 0 being the present, let y(t) be the revenue per day at time t. So, we know y(0), the current revenue. Let b be the revenue per day of the full, planned fleet and business.

Then at time t, the number of customers talking is proportional to y(t) and the number of the rest of the customers listening is proportional to (b - y(t)).

So, for some constant of proportionality k, we have that the growth rate in revenue per day is

d/dt y(t) = y'(t) = k y(t) (b - y(t))

where we have y(0) and b.

So, this is an initial value problem for a first order, linear, ordinary differential equation, but just freshman calculus is plenty to solve it in closed form. Sure, the solution has some exponentials. I'll post the solution later on request!

So, one day Senior VP Planning Mike Basch and I picked a value of k that yielded a reasonable graph, and I drew the graph. That was Friday. The BoD meeting was the morning, Saturday, with Mike traveling. At noon I got a call from Senior VP Roger Frock asking if I knew about the revenue projections and could I come to the BoD meeting? I did and did. When I got there, the graph was on a table, and our two representatives of BoD member General Dynamics were standing in the hall with their bags packed. No one was happy. Roger pointed to a few places on the graph, and I used my HP calculator to reproduce the points. Everyone got happy.

Later I learned that the graph had been presented to the BoD early in the meeting; the General Dynamics guys asked how the graph was calculated; all the FedEx people at the meeting tried to learn how; near noon the Dynamics guys gave up on FedEx, got plane tickets back to Texas, went to their rented rooms and packed, and as a last resort returned to the BoD meeting and were standing in the hall when I arrived. With my results, they unpacked and stayed. Had they left, FedEx would have ended.

So, just a little calculus saved FedEx, and since I was the only one around who knew still knew calculus I was the only one who understood the solution.

The projections were a smooth curve that rose slowly, rose more quickly, had an inflection point, rose less quickly, and became asymptotic from below at b. So, the curve was a lazy S.

I suspect that the curve is the famous logistic curve also important in, say, the elements in neural networks and has long been seen as a good, first cut description of the growth of new products, e.g., TV sets when they were new.

With the way I derived the curve, it is an axiomatic approach to viral growth or word of mouth market growth.

There is lots of nice stuff that can be done with calculus.

For more with calculus, I would recommend famous texts by Apostol, Rudin, Royden, and Rudin again, through measure theory and then texts by Loeve, Breiman, Neveu, and Chung for the connections with probability. The third edition of Rudin's Priciples does very well with the modern Stokes theorem, exterior algebra of differential forms, and the inverse and implicit function theorems. The first half of Rudin's Real and Complex Analysis does well with measure theory and introductions to functional analysis. The Apostol text is Mathematical Analysis: A Modern Approach to Advanced Calculus -- it's NOT very "modern" but gives a lot of what need to know to read mathematical physics, especially Maxwell's equations. For stochastic processes, Doob, Karatsas and Shreve. For Princeton and its department of Operations Research and Financial Engineering, the more elementary parts of stochastic optimal control via, say, Nemhauser, Dreyfus and Law, Bertsekas and Shreve, Dynkin and Yushkevich, etc.

thread

graycatApr 2018

I was proposing how to make progress in understanding some of the data manipulations being done now by what appears to be most of ML/AI. That is, for the data manipulations, get those from the consequences of some theorems. For an example, apparently regression is important in current ML/AI: Well, going way back, 50+ years, we have a lot of solid math for regression.

Then I mentioned that from Euclid through calculus to wave equations, there's more math from theorems and proofs we can and sometimes do apply. And we can stir up still more applicable math.

My goal was just suggesting how to do better with real, valuable applications to important real world problems. We have such examples from US national security -- the A bomb, the H bomb, GPS, stealth, phased array radar, adaptive beam forming sonar, etc.

By analogy, I was suggesting a better socket wrench or numerically controlled milling machine and not a self-driving car. I was not suggesting anything that we could regard as intelligent, not even as intelligent as a field mouse.

IMHO, powerful math with valuable applications is now very doable. We can schedule equipment maintenance, airline crews, airline fleets, workers, trucks, etc. We can have computers do the data manipulations specified by the math and have the Internet move the data.

But for also having systems that are as intelligent as a field mouse, kitten, puppy, octopus, song bird, even walk as well as a cockroach, that's harder.

At one time I worked in some AI based on some MIT AI work; I wrote software, worked with GM Research, gave a paper at the Stanford AAAI IAAI conference, as sole author or co-author published a list of papers. Here I was not suggesting anything that has anything to do with any of the MIT AI work I've known about.

For people at MIT who have done work more like I have in mind, I can think of D. Bertsekas and M. Athans.

Athans was in deterministic optimal control, and the OP mentioned that that field did "back propagation" long ago. Athans is, was, a good applied mathematician. When I was at FedEx, I chatted with him in his office on how to find how best to climb, cruise, and descend airplanes. He told me a cute story about how an F-4 could get minimum time to climb, say, IIRC, to 100,000 feet: Climb up to just 5,000 feet or so, go into a dive, get supersonic where actually the drag was less, and then go nearly vertical, all supersonic, directly to 100,000 feet.

For neural networks, IIRC there is some nice math that shows how general those can be for representing functions, and for some parts of stochastic optimal control Bertsekas proposed such a use for neural networks. There he, as usual, was being mathematical.

thread

graycatMar 2017

I can accept Newton's second law of motion and Einstein's general relativity, but so far I'm less accepting of deep learning for finding some coefficients for fitting some data or neural networks for fitting some sigmoid functions to some data. Apparently the sigmoid functions have some nice theoretical properties and can be useful in approximating some complicated data; e.g., IIRC D. Bertsekas at MIT used such fitting to approxmate the optimal value functions in stochastic optimal control. Or, have some big data and a lot of computing cycles? One way to soak them up is stochastic optimal control, possibly with some sigmoid fitting. To me, that's more promising for the future than what I've seen in the current AI based on Breiman, etc., and I'm a really big fan of Breiman.

What is going on in such cases -- learning, neural -- is at best some applied math, but the learning, neural, and intelligence are trying to suggest, hint, hint, that the math is somehow close to what intelligent humans with biological brains do.

The effects of the hints are strong enough to give lots of newsies another of their favorite means of getting eyeballs and clicks, that is, there are big threats, the sky is falling, nearly all humans will lose their jobs, and the robots might take over as in some movies.

Yes, somewhere in a buried paragraph there maybe an admission that the learning is likely nothing like what humans, kitty cats, puppy dogs, whales, or crows do, but that disclaimter, if want to call it that, is hardly a drop of water on the hype inferno.

At one time some IBM publicity got some housewives to believe that iBM's computers were "gigantic electronic human brains".

I know; I know; such hype is not the worst problem in the world and, really, is affecting me hardly at all. But when I was in an AI group at IBM's Watson lab, still I was embarrassed to say artificial intelligence. I knew, yes, that the meaning was supposed to be just a computer doing something that would require human intelligence if a human did it, e.g., play chess well, but that the approach of the AI chess, a big tree search with alpha-beta pruning, etc., was nothing like what humans did playing chess. But with this definition some 0-1 integer linear programming for scheduling at an airline should be far above merely intelligent since it is far above what an unaided human could do -- although humans invented the applied math that is 0-1 integer linear programming.

I call the crucial core technology for my startup not AI but just some applied math. The world is awash in good applied math -- statistical hypothesis testing, the fast Fourier transform, numerically exact matrix inversion (number theory techniques), Wiener filtering, deterministic optimal control, and hundreds more -- all without a single claim of anything intelligent in any of the books, papers, courses, etc.

thread

NY_USA_HackerMay 2011

If you want something written as well as Halmos, then start writing, and good luck!

"Historical accident": Well, sure, in part nearly all the pure math departments pushed out any such topics!

You are correct: Optimal control has been mostly in advanced parts of electrical engineering.

So, optimal control in EE was an example doing math outside math departments. Of math done outside math departments, control theory is relatively good mathematically.

And, yes, there should be applications elsewhere.

For physics, yes, the deterministic theory of optimal control has been seen as replacing the older calculus of variations which goes back to Newton.

My references are old.

More recent work on stochastic optimal control has been by R. T. Rockafellar at University of Washington.

For

Stuart E. Dreyfus and Averill M. Law, 'The Art and Theory of Dynamic Programming', ISBN 0-12-221860-4, Academic Press, New York.

'dynamic programming' is a big part of the discrete time versions of optimal control. It can be stochastic or deterministic. The linear-quadratic-Gaussian (linear 'plant' or system, quadratic cost to be minimized, and Gaussian exogenous random variables) has 'deterministic equivalence' -- nice -- and this book treats it. The book is a good, elementary start. Apparently Dreyfus was a R. Bellman student.

For

L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mischenko, 'The Mathematical Theory of Optimal Processes', ISBN 0-470-69381-9, Interscience Publishers, John Wiley & Sons, New York.

it was, of course, the main source of the Pontryagin maximum principle, and, thus, a swift kick in the back side for parts of US aerospace in the 1960s.

For

Michael Athans and Peter L. Falb, 'Optimal Control: An Introduction to the Theory and Its Applications', McGraw-Hill Book Company, New York.

Athans was long in EE at MIT and did some military work, e.g., on parts of the C5A airplane. Falb was at Brown's Division of Applied Mathematics.

One Athans story was the 'control' for least time to climb to, say, 100,000 feet for an F-4: Go up to a few thousand feet, go into a dive, get supersonic, get the lower drag of a few hundred knots above Mach 1, and then with the lower drag continue supersonic to the final altitude. I was not able to know if that 'control' idea was just intuitive or directly from computation and the Pontryagin maximum principle which is, after all, just a necessary condition, i.e., local optimality.

Also in that division at Brown, see the papers of Harold Kushner. As I recall, he wrote out a stochastic version of the Pontryagin maximum principle.

For

E. B. Dynkin and A. A. Yushkevich, 'Controlled Markov Processes', ISBN 0-387-90387-9, Springer-Verlag, Berlin.

there are connections with economic planning.

For

Dimitri P. Bertsekas and Steven E. Shreve, 'Stochastic Optimal Control: The Discrete Time Case', ISBN 0-12-093260-1, Academic Press, New York.

the math is done carefully. So there is a lot of attention to measurability. Part of the reason is the issue of 'measurable selection': This can be a deep subject, but often a relatively simple way out is via regular conditional probabilities as in

Leo Breiman, 'Probability', ISBN 0-89871-296-3, SIAM, Philadelphia.

For

David G. Luenberger, 'Optimization by Vector Space Methods', John Wiley and Sons, Inc., New York.

this has likely the easiest mathematical treatment of deterministic optimal control and also Kalman filtering and also the math needed for 'least action' in physics.

For

E. B. Lee and L. Markus, 'Foundations of Optimal Control Theory', ISBN 0471-52263-5, John Wiley & Sons, New York.

this tried to be clean mathematically when it was written. At the beginning, should know some relatively advanced results in ordinary differential equations, e.g., as in

Earl A. Coddington and Norman Levinson, 'Theory of Ordinary Differential Equations', McGraw-Hill, New York.

For more, you can consider non-linear filtering and connections with mathematical finance.

The above is just from my bookshelf. Likely now a better bibliography could be assembled via the Internet which actually is at least a little larger than my bookshelf!

thread