Joe Damato writes deep technical posts on Linux internals, Ruby VM performance, debugging, and binary formats.
Commenters call his posts fascinating and say they show the value of understanding what's under the hood, though he posts infrequently.
Garbage collection is a particularly cross-cutting implementation issue, and tends to interfere with all kinds of abstraction. You're essentially making a system that knows which specific ways abstraction can be broken safely, so that you can defer resource management to it in the future. Retrofitting a GC is even harder. You can bolt on a conservative collector (like boehm-gc), sure, but that can lead to many unexpected performance issues (see e.g. http://timetobleed.com/6-line-eventmachine-bugfix-2x-faster-gc-1300-requestssec/).
Writing a virtual machine for a GC'd language and then trying to give it a portable, reasonably convenient C API convinced me that it's a significantly harder issue than most people realize. (Particularly after scrapping my third prototype!) The C API also impacts many things, and tends to bring up nasty edge cases in the GC.
Lua seems to have settled on a design that works unusually well, and is worth studying. Still, Lua can safely cut corners that Python and Ruby cannot - they have threads, which also break a lot of assumptions.
Joe Damato has a great albeit infrequently updated programming blog, http://timetobleed.com/ . I think he wrote the memprof ruby gem. Posts HN may really enjoy:
an obscure kernel feature to get more info about dying processes [1]
a presentation from some ruby conf, particularly slide set 2 which details how memprof works and talks about the abi, etc [2]
plus a bunch of discussion of profiling tools to look at exactly what gcc or your vm of choice are doing. Highly recommended.
[1] http://timetobleed.com/an-obscure-kernel-feature-to-get-more-info-about-dying-processes/
[2] http://timetobleed.com/slides-from-mwrc-2010/
The first link mentioned in the article was a very interesting read.
http://timetobleed.com/defeating-the-matasano-c-challenge-with-aslr-enabled/
It's a fairly clear explanation. Before the language warriors come out in force, I'll mention that it appears that all of the problems turn out to be issues with 1) calling new and 2) C-style casts.
Exploiting the issue requires someone with a detailed knowledge of C++ inner workings. It would have taken me a long time to figure it all out.
Avoiding the issue? Well, most modern C++ doesn't look very much like the problem code for very good reasons.
One of the biggest problems with this style of "scan all memory" GC is that you have to stop all threads (flush all pointers to memory) to perform a collection. Pointers held in registers in other active threads are inaccessible to the GC thread.
Failing to stop the world (or ensure all pointers are in memory rather than registers) can result in live objects being collected. This happened to Ruby at one point: http://timetobleed.com/the-broken-promises-of-mrireeyarv/
Edit: This boils down to breaking the compiler's model of memory as specified by the C standard; all bets are off.
I don't know why people advise others to splat their compiled 3rd party software all over /usr/local when /opt is there specifically for that purpose.
Also, enabling pthread on 1.8 has some significant performance problems unless you're using REE. For more reading on that:
http://timetobleed.com/fix-a-bug-in-rubys-configurein-and-get-a-30-performance-boost/
If you've never worked with Joe, this is the kind of nonsense he does in his sleep.
I first worked with Joe four years ago. He joined the team knowing no Ruby. What did he do his first week? Diagnose and patch a pretty big bug in MRI's threading implementation.
His blog is equally intense: http://timetobleed.com