Skip to content
HomeVideos
AllLongsCategoriesPlaylists
BooksBlogsPodcasts
HistoryListsWatch laterLiked videosFollowing

The videos, books, blogs and podcasts Hacker News keeps recommending, all in one place.

VideosBooksBlogsPodcastsAboutPrivacySettings

Data from Hacker News, HackerTube, HackerBooks, HackerBlogs and Open Library. Credits · Not affiliated with Y Combinator.

HomeVideosBooksBlogsPodcasts
Ctrl K

    Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

    Sasha Rush

    3 HN recommenders · recommended in 3 months, 2024

    YouTube

    39:43 · 35K views · Jul 5, 2023

    Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

    HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

    AI, Machine Learning & DataMachine learning & deep learningLLMs & generative AI
    1. 27:07
      Attention Is All You NeedYannic Kilcher814K views • 8 years ago
    2. 1:22:38
      CS480/680 Lecture 19: Attention and Transformer NetworksPascal Poupart374K views • 7 years ago
    3. 28:48
      LSTM is dead. Long Live Transformers!Seattle Applied Deep Learning541K views • 6 years ago
    4. 27:14
      Transformers, the tech behind LLMs | Deep Learning Chapter 53Blue1Brown12M views • 2 years ago
    5. 3:31:24
      Deep Dive into LLMs like ChatGPTAndrej Karpathy10M views • 1 year ago
    6. 59:48
      [1hr Talk] Intro to Large Language ModelsAndrej Karpathy4.3M views • 2 years ago
    7. 1:31:13
      A Hackers' Guide to Language ModelsJeremy Howard580K views • 3 years ago
    8. 26:10
      Attention in transformers, step-by-step | Deep Learning Chapter 63Blue1Brown4.9M views • 2 years ago
    9. 42:40
      State of GPT | BRK216HFSMicrosoft Developer786K views • 3 years ago
    10. 8:25
      Large Language Models from scratchGraphics in 5 Minutes375K views • 4 years ago
    11. 1:56:20
      Let's build GPT: from scratch, in code, spelled out.Andrej Karpathy8.2M views • 3 years ago
    12. 20:39
      AI Language Models & Transformers - ComputerphileComputerphile354K views • 7 years ago
    13. 48:32
      Sparks of AGI: early experiments with GPT-4Sebastien Bubeck1.8M views • 3 years ago
    14. 2:13:35
      Let's build the GPT TokenizerAndrej Karpathy1.3M views • 2 years ago
    15. 1:57:45
      The spelled-out intro to language modeling: building makemoreAndrej Karpathy1.3M views • 4 years ago
    16. 1:19:08
      CS231n Winter 2016: Lecture1: Introduction and Historical ContextAndrej Karpathy463K views • 10 years ago
    17. 1:03:32
      John Schulman - Reinforcement Learning from Human Feedback: Progress and ChallengesUC Berkeley EECS88K views • 3 years ago
    18. 1:11:41
      Lecture 1 | Natural Language Processing with Deep LearningStanford University School of Engineering821K views • 9 years ago
    19. 2:25:52
      The spelled-out intro to neural networks and backpropagation: building microgradAndrej Karpathy4.4M views • 4 years ago
    20. 36:02
      ChatGPT with Rob Miles - ComputerphileComputerphile517K views • 3 years ago

    3 comments from Hacker News

    1. @a_wild_dandanMar 2024· in “Jamba: Production-grade Mamba-based AI model”

      To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: this video

      Strongly recommendsView on HN
    2. @srushFeb 2024· in “The Annotated S4”

      Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: this video Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).

      RecommendsView on HN
    3. @anomaly72Jan 2024· in “MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts”

      Would recommend Sasha Rush's lecture this video , it provides the intuition for state space models via linear RNNs

      RecommendsView on HN