Episode
Recurrence and Attention for Long-Context Transformers with Jacob Buckman - #750
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) · Oct 7, 2025 · 57 min
Today, we're joined by Jacob Buckman, co-founder and CEO of Manifest AI to discuss achieving long context in transformers. We discuss the bottlenecks of scaling context length and recent techniques to overcome them, including windowed attention, grouped query attention, and latent space attention. We explore the idea of weight-state balance and the weight-state FLOP ratio as a way of reasoning…