Jack Gallant
1 HN recommender · recommended in 11 months, 2013–2024
0:29 · 6.8M views · Sep 22, 2011
Jack Gallant's lab demonstrates reconstructing movie trailer clips a subject watched from fMRI brain activity, using regression models trained on hours of viewing data. The video shows the original clip beside the reconstruction.
HN cites it as the landmark brain-decoding demo and compares newer reconstruction work against it.
15 comments from Hacker News
This is a much cooler example this video If I'm understanding the description correctly, they are just training it to recognize what the image is closest to and taking an slice of a youtube video that most closely matches it. I imagine if they used a more efficient method or trained it more, they could do way better. It seems like most of the data to build an accurate picture of what they are seeing is already there.
Although it's hard to tell from the images presented with the article, the face generation looks like it could be similar to the techniques used in Nishimoto et al., 2011, which used a similar library of learned brain responses, though for movie trailers: this video Their particular process is described in the YouTube caption: The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows: [1] Record brain activity while the subject watches several hours of movie trailers. [2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured. (For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.) [3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions. [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Se
According to the abstract: Here, we present a neural decoding approach in which machine learning models predict the contents of visual imagery during the sleep onset period given measured brain activity, by discovering links between human fMRI patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. So it sounds like the fMRI patterns are associated with words via verbal reports from humans, and the video is simply generated from image databases by using those words. In other words, they classify the fMRI patterns based on how well they correspond to various words (measured while the subjects are awake and giving verbal descriptions of images they're looking at), and then the program measures the subjects' fMRI patterns while they're dreaming, determines the strongest 'word signals' in the patterns, and selects images from a photo database corresponding to those words and assembles them into a video based on how strong the similarity score for each word is at each moment in time. It's interesting, but the video seems like more of a "reenactment" of a dream than a prediction. It doesn't look like you can infer actions and occurrences within a dream from this. But this is all just my interpretation. edit: This is a fundamentally different approach from the one used in other research (e.g. this video me
I feel like we are so close to wirelessly monitoring all brain waves (next 50 years) we will probably be able to, in real time, recognize and diagnose abnormal brain patterns. I think it would be promising if we get this sensor tech up and then perhaps deploy nanotech to interfere with maladaptive brain patterns. The ethics of such a device is considerably sticky. TED talk by Jack Gallant [LINK] recreating vision: this video another ted talk by John-Dylan Haynes [LINK] paraphrased: "wireless non-contact mind reading tech may be cheap and mass producible within the next 50 years." the only rate limiting step with his method of decoding thoughts is that it takes time under a high energy device to pair associations in the brain. Think of how much screen time we currently use. If we could pair a wireless 'headset' like headphones while looking at pictures on a screen it would be possible to build a brain bank of children->tenenagers. Currently we just need a lot of processing power and reserved bandwidth on the internet to do some cool stuff.
The errors and bias in these systems add to the ethical concerns. While it is arguably a good thing to substitute torture with brain image interrogation, there is a risk of putting too much trust in these systems, subjecting innocents to days of leading investigation, just because the computer said there is something to be found there. Israeli airport security (arguably the best in the world) deploys derivatives of these systems, that look at micro-gestures, elevation of heart rates, pupil dilation, and temperature changes, to see if passengers respond with familiarity to terrorist imagery flashed on a screen as they walk by it. If that already works in practice, imagine the same, but being strapped with hundreds of sensors. See also the 2010 research on image reconstruction from brain activity, and extrapolate that 10 years in the future and applied to military interrogation: this video