Episode
Speculative Decoding and Efficient LLM Inference with Chris Lott - #717
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) · Feb 4, 2025 · 1 h 17 min
Today, we're joined by Chris Lott, senior director of engineering at Qualcomm AI Research to discuss accelerating large language model inference. We explore the challenges presented by the LLM encoding and decoding (aka generation) and how these interact with various hardware constraints such as FLOPS, memory footprint and memory bandwidth to limit key inference metrics such as…