Episode
Generative Benchmarking with Kelly Hong - #728
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) · Apr 23, 2025 · 54 min
In this episode, Kelly Hong, a researcher at Chroma, joins us to discuss "Generative Benchmarking," a novel approach to evaluating retrieval systems, like RAG applications, using synthetic data. Kelly explains how traditional benchmarks like MTEB fail to represent real-world query patterns and how embedding models that perform well on public benchmarks often underperform in production. The…