Stefan Marr, a computer science researcher, writes about language implementation, interpreters, meta-compilation, concurrency, and benchmarking methods.
HN commenters point to his group's research on JIT and interpreter performance, and link his posts on bytecode versus AST interpreters and the cost of safety in Java.
> Or, in other words, the performance difference comes somewhere else than the choice of programming language.
We're comparing the code generated by a compiler or VM/JIT of a specific programming language, not the programming language itself. I'm amazed that this is even up for discussion in the article. It is clear that even with the same language there can be differences in implementation. Here is e.g. a measurement series I did to compare different Smalltalk implementations (LjSOM is mine): http://software.rochus-keller.ch/are-we-fast-yet_crystal_lua_node_som_pharo_i386_results_2020-12-29.pdf. Smalltalk is a tough nut to crack because nearly all statements are closures; the Open Smalltalk VM team invested two decades to achieve a decent performance, but it's still quite behind e.g. V8 or a statically typed AOT compiled language. The Are-we-fast-yet suite (https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/) gives more credible results than CLBG (referenced in the article) from my point of view because of the stricter rules the implementations have to adhere to (in favour of fair comparisons) and the larger benchmark apps. In return, the implementation gives significantly more work, but it is doable and worth it; here is one of my implementations that helped me significantly improve the compiler (still in progress): https://github.com/rochus-keller/Oberon/tree/master/testcases/Are-we-fast-yet; this also demonstrates that -…
We compared the runtime performance of GraalSqueak with OpenSmalltalkVM (formerly Cog) and RSqueak (RPython-based VM) in Figure 4 of our MPLR’19 paper [1] for the first time. Since then, we've worked on a couple of performance optimizations. So at the moment, I think GraalSqueak is only slower in the DeltaBlue benchmark... in all others, it's (often significantly) better than OpenSmalltalkVM. However, please also note its limitations which we discussed in section 5.2, especially with regard to Smalltalk's interrupt handler and Partial Evaluation performed by the Graal compiler.
So far, I'd say our observations wrt RPython match the ones discussed in "Tracing vs. Partial Evaluation" [2]: Language implementers have to do more work in Truffle, but probably get better peak performance in return (not talking about warmup or memory consumption here).
RE "a more optimized bytecode" set: Sista [3] (an extension of the OpenSmalltalkVM) is doing something like that, too. I'd guess performance would be more or less the same using GraalVM as Truffle produces highly specialized code. But, of course, this would need to be benchmarked. The advantage of the Sista approach is that it's managed on the image level, so specialized versions of methods are persisted as part of the image. AFAIK the GraalVM team is working toward persisting compiled code caches, which is kind of similar but on the level of the language implementation framework.
Hope this answers your questions!
[1]…
It's interesting to read the OP's response to partial evaluation (employed by Truffle) and why he switched to meta-tracing in RPython. Basically, the inlining/specialization decisions in PE-based JITs were harder to control than just letting the VM observe loop iterations.
Here's a paper regarding a pair of language implementors' experience using both techniques:
http://stefan-marr.de/papers/oopsla-marr-ducasse-meta-tracing-vs-partial-evaluation/
By changing the abstract syntax tree during execution, type-generic nodes in the AST can be replaced with type-specialized nodes. This carries more overhead than tracing through the interpreter. It also requires more explicit work from the user to achieve optimized results.
The authors of the above study conclude that the resulting performance was similar, but that meta-tracing was an easier technique to use than PE. This agrees with the OP's assessment.
> Just fix the VM then
No the point isn't that the VM is broken. The point is that the Ruby language has semantics and is used in idioms that nobody knows how to optimise well yet.
> tinyrb based on potion based on lua with classes is ~200x faster in method calls
tinyrb is interesting, but it just isn't the same semantics as Ruby:
https://github.com/macournoyer/tinyrb#what-wont-be-in-tinyrb-tiny-patches-accepted
tinyrb doesn't have a better VM - they're implementing a different programming language with all the hard bits left out!
In my project we've been having to do new original research into how to optimise Ruby - introducing concepts such as dispatch chains which haven't been needed more because people don't try to push metaprogramming in most language like they do in Ruby.
http://stefan-marr.de/downloads/pldi15-marr-et-al-zero-overhead-metaprogramming.pdf
Particle systems [1].
AR. VR [2][3][4][5].
Image processing, neural nets.
But not in programming languages like C, Java or Linux.
Only in massively parallel message passing languages[4].
I suggest a Smalltalk message passing scalable cluster VM like the RoarVM [6][7][]8].
[1] Shadama 3D. Yoshiki Oshima. https://www.youtube.com/watch?v=rka3w90Odo8
[2] OpenCobalt Alpha. https://www.youtube.com/watch?v=1s9ldlqhVkM&t=13s
[4] OpenCroquet Alan Kay Turing lecture. https://www.youtube.com/watch?v=aXC19T5sJ1U?t=3581
[5] OpenCroquet Alan Kay et al. https://www.youtube.com/watch?v=XZO7av2ZFB8
[6] RoarVM paper. David Ungar et al. https://stefan-marr.de/renaissance/ungar-adams-2009-hosting-an-object-heap-on-manycore-hardware-an-exploration.pdf
[7] RoarVM Demo. https://www.youtube.com/watch?v=8pmVAtk612M
[8] RoarVM Demo. https://www.youtube.com/watch?v=GeDFcC6ps88
There aren't many people looking at these JITs at the moment. Stefan Marr[1]'s group[2] is, I believe, the where most of the research is currently done. A recent paper[3] compares performance of interpreters in RPython and Graal. Their baseline performance is Java, and they achieve performance close to V8, which itself is about 2x slower than Java.
My summary is you can write fast interpreters + get JIT for free, but fast JIT for dynamic languages still means 2x slower than JIT for statically typed languages (and Java definitely leaves some performance on the table due to how it represents data).
[1]: https://stefan-marr.de/
[2]: https://research.kent.ac.uk/programming-languages-systems/
[3]: https://dl.acm.org/doi/10.1145/3622808