MLSS Iceland 2014
3 HN recommenders · recommended in 3 months, 2017–2024
1:29:43 · 48K views · Nov 17, 2014
Michael Betancourt's lecture at MLSS Iceland 2014 on efficient Bayesian inference with Hamiltonian Monte Carlo. It covers MCMC concepts and motivation and the problems of sampling in high-dimensional distributions.
HN people return to it as a clear primer on MCMC and HMC and on why high dimensions behave counterintuitively.
3 comments from Hacker News
Recently watched this on Hamiltonian Monte Carlo, and it was a fantastic primer on the basic concepts and motivations of mcmc, and later hmc this video
Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: [LINK] - Thinking outside the 10-dimensional box - 3blue1brown: [LINK] - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: this video
This is an interesting take and kudos to the author for relaying a helpful way to think about high dimensional distributions. I really like and often come back to this talk by Michael Betancourt where the theme is quite similar: this video