Hazy Research Blog

A medium-spicy take of mine is that a bytecode VM in a GPU kernel is not as bad of an idea as one might think, and in some cases it can actually be the most reasonable solution. Some fun examples:

1. As mentioned in the post above, the Dolphin emulator famously implements the entire Gamecube/Wii GPU pipeline in a single gigantic ubershader, and this is useful because it avoids shader compilation stalls [1].

2. Blender's Cycles renderer implements its shading graph eval system as a bytecode VM in a GPU kernel [2]. IIRC early versions of Vray GPU did something similar. There are better ways of course, but a VM gets you surprisingly far as a general approach.

3. Finally, a lot of ML frameworks (Tensorflow, PyTorch, etc) by default use the GPU relatively suboptimally (especially without kernel fusion and such). Tensor frameworks can extract a lot more perf out of GPUs using a VM-in-a-giant-kernel approach [3].

If you think abstractly about how a GPU SM actually works (using CUDA terminology here), all threads in a warp must execute in lockstep and the cost of execution divergence across threads in a warp is that you effectively run serially, losing the parallel advantage of the SM. This penalty gets magnified enormously if you are doing memory reads after wherever the execution divergence happens, since you now have multiple slow memory stalls in serial instead of one big memory read at once for all threads. If you're clever about implementing a bytecode VM, you can load as…

Here's a list of tools for scaling up transformer context that have github repos:

* FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention

* Heinsen Routing: In my experience, the current best solution for n×m attention, i.e., mapping n tokens to m tokens. It's like a souped-up version of attention. I've used it to pull up more than a million tokens as context. Memory use and compute are O(nm). It works, but in my (limited) experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: https://github.com/glassroom/heinsen_routing

* RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attention in transformers. In my (limited) experience, it doesn't. Others seem to agree: https://twitter.com/arankomatsuzaki/status/1639000379978403853 . Code: https://github.com/BlinkDL/RWKV-LM

* RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: https://github.com/booydar/t5-experiments/tree/scaling-report

In addition, the group that developed FlashAttention is working on state-space models (SSMs) that look promising to me. The idea is to approximate n² attention dynamically using only O(n log n) compute. There's no code…

We (a few PhD students at Stanford Hazy Research) just released an open-source Python library called [Meerkat](http://meerkat.wiki) where we built an interactive data frame powered by foundation models to wrangle unstructured data (images, videos, text docs...).

Unstructured data is rich, diverse, complex, and pervasive which also makes it incredibly challenging to visualize, interact with, and use. We think large foundation models (FMs) like GPT-3 or CLIP will change this by allowing us to directly operate on such data. FMs are improving rapidly in their ability to read, interpret and generate unstructured text, images, even videos.

The main problem is that FMs need to be controlled, and made more reliable as a software abstraction. Right now, it's like you're on a ship in a storm, and you're fighting the waves without a steering wheel. We felt that a lot of existing tools (think pandas or even Streamlit) were tailored to structured data, and weren't really well suited to the new challenges that are emerging in this paradigm.

We built Meerkat to help. Meerkat systematically combines data frames with FMs and interactivity in a way that preserves the flexibility of FMs with added control. We aren't trying to reinvent the wheel, and we're hoping that we can lean on familiar interfaces like data frames while bringing something new to the table.

If you're interested in details, you can read more about Meerkat on our research blog:…

Research suggesting that much of the power of the transformer architecture comes from associative recall over long sequences that does not require scaling model dimensions. They design state space models that narrow the gap.

Overview https://hazyresearch.stanford.edu/blog/2023-12-11-zoology0-intro

Zoology 2 https://hazyresearch.stanford.edu/blog/2023-12-11-zoology2-based

Monarchs and Butterflies: Towards Sub-Quadratic Scaling in Model Dimension. Scaling in model dimension as opposed to sequence dimension scaling in the previous posts. https://hazyresearch.stanford.edu/blog/2023-12-11-truly-subquadratic

I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabilities in the last year - but I don't think it's ultimately how we'll get to AGI.

[0] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

Scaling if context window size has been a problem but now good potential of solutions using mamba.

HyenaDNA is one to look at wrt DNA.

And here are some other interesting links from Erik Garrison—a leader in pangenomics.

https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna

https://github.com/instadeepai/nucleotide-transformer

https://dl.acm.org/doi/pdf/10.1145/3535508.3545512

https://github.com/dnbaker/bioseq

https://huggingface.co/AIRI-Institute/gena-lm-bert-base

https://discuss.huggingface.co/t/dna-long-sequence-tokenization/16154