Sakana AI

Hi HN,

I am one of the authors from Sakana AI and MIT. We just released this paper where we hooked up LLMs to the classic 1984 programming game Core War. For those who haven't played it, Core War involves writing assembly programs in a language called Redcode that battle for control of a virtual computer's memory. You win by crashing the opponent's process while keeping yours running. It is a Turing-complete environment where code and data share the same address space, which leads to some very chaotic self-modifying code dynamics.

We did not just ask the model to write winning code from scratch. Instead, we treated the LLM as a mutation operator within a quality-diversity algorithm called MAP-Elites. The system runs an adversarial evolutionary loop where new warriors are continually evolved to defeat the champions of all previous rounds. We call this Digital Red Queen because it mimics the biological hypothesis that species must continually adapt just to survive against changing competitors.

The most interesting result for us was observing convergent evolution. We ran independent experiments starting from completely different random seeds, yet the populations consistently gravitated toward similar behavioral phenotypes, specifically regarding memory coverage and thread spawning. It mirrors how biological species independently evolve similar traits like eyes to solve similar problems. We also found that this training loop produced generalist warriors that were robust even…

Note that this paper appears to have a critical flaw: they didn't actually verify that their "optimized" kernels have the same output as the original kernels.

So some of their "optimized" kernels gamed the system by computing only parts of the actual result.

Look at their most "improved" (147x faster!) kernel for example: https://pub.sakana.ai/ai-cuda-engineer/kernel/1/15/optimize-b5-s4-e1-sweep/3/2/1/strided_efficient_triangular_mm_edit_1

If you read the code carefully, and in particularly the kernel launch parameter, you will see that this only computes the first row of the matrix. Which means it's generating incorrect results.

As I write this (after the updates to the evaluation code), https://pub.sakana.ai/ai-cuda-engineer/kernel/2/23/optimize-b10-s4-e0-sweep/7/3/0/fused_ops_strided_optimized_base is on their top of their list of speedups, with a claim of 128x speed up on a fused 3D convolution + groupnorm + mean.

The generated implementation doesn’t do a convolution.

The 2nd kernel on the leaderboard also appears to be incorrect, with a bunch of dead code computing a convolution and then not using it and writing tanhf(1.0f) * scaling_factor for every output.