Joe Armstrong

I think Joe Armstrong's thesis:

https://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf

is an interesting read and provides a more concrete example of how to run a highly concurrent and fault tolerant application.

It doesn't talk about the social or psychological bullet points in this article, it is more technical. But I found it very readable.

As an addition, I can think of these patterns (just thinking about it in 1 minute, mostly remembering Erlang talks I've listen to, some from practice):

* Build system out of isolated components. Isolation will prevent failures from propagating. In Erlang just launch a process and don't use custom compiled C modules loaded in the VM. In other cases launch an OS process (or container).

* If your service is running on one single machine, it is not fault tolerant.

* Don't handle errors locally. Build a supervision tree where some of part of the system does just the work it is are intended to (ex.: handling a client's request), and other (isolated part) does the monitoring and error handling. Have one process monitor others, one machine monitor another etc.

Once a segfault or malloc fault has occurred installing a handler and trying to recover might not be the best solution. Restarting an OS (or Erlang) process might be easier. Another way to put it, once the process has been fouled up, don't trust it to heal itself. Trust another one to clean up after it and spawn a new instance it.

* Try not to have a master or a single point…

btw, HOP uses the Bigaloo web server and so isn't quite adequate apparently for a high traffic site (like I assume most here will have). here's an email exchange I had with the author the other day- sorry if the comment's a bit big:

I apologize for such a late answer. I'm always fighting with email. I'm
definitively not good at it because I'm not able to answer at a decent pace.

Hi Manuel. I've just been playing with Hop. Well done. It's just
beautiful. A couple of questions about it:
first, the bigloo web server- is this completely necessary? I guess one
of 4 things would need to happen for hop to be totally ideal:
Bigloo implements light-weight threads Gambit-scheme style so that it
can have erlang-like load capacity (
http://www.sics.se/~joe/apachevsyaws.html ) (most preferable option,
IMO, though I have no idea what your feelings are here. To me this
would make bigloo light-years ahead of all other scheme imps.)
I have tried several things concerning the multi-threading. You can read
our FairThreads proposal (http://www.inria.fr/mimosa/Manuel.Serrano).
When I did this study I though that FairThreads was a decent solution.
Now that's no longer clear to me. Hop does not use Fair threads but
Posix like threads. I'm quite aware of all the troubles that come with
this kind of threads. I have not considered an Erlang-like solution
yet but that's something I could envision.

Get hop to work with a gambit-scheme server instead of bigloo (where
would i start with…

There are other languages you could make it small in, but those languages probably aren't among them.

Haskell: Hugs 98 is 723K, but it depends on /usr/lib/hugs, which is 7.3M. GHC can almost do it; I compiled a "hello, world" program with GHC and it "only" came out to 366K, which is 84K bigger than μTorrent's reported size. With UPX, however, it packs to 130k. (ocamlopt's tax is in the same range.) Maybe Haskell would be an option? Or does it bloat up when you include the libraries you'd need for this?

Erlang: My Erlang install is 97M. I don't know how much of that is minimally necessary to run things, but /usr/lib/erlang/erts-5.6.3 is 3.6M, 12 times the size of μTorrent. Stand Alone Erlang (http://www.sics.se/~joe/sae.html) is 1.5MB compressed.

Common Lisp: All the Common Lisp implementations are pretty big (SBCL, say, is 53M, of which 25M is the image sbcl.core), and they can't produce a standalone executable without including most of the environment. So you're starting at 25M.

Java: The JRE on my system is 98 megabytes, so a Java application to run on Win32 will be 98 megabytes plus the application code. Even if you compile your app with GCJ, libgcj is 33 megabytes.

Now, you could argue that the size of this stuff isn't relevant because it isn't specific to your hypothetical Common Lisp implementation of BitTorrent, but could also be used by whatever other Common Lisp programs the user has installed. But you would be treating a very-low-probability event as if it…

To quote Joe Armstrong: "As Erlang became popular we were often asked "Is Erlang OO" - well, of course the true answer was "No of course not" - but we didn't to say this out loud - so we invented a serious of ingenious ways of answering the question that were designed to give the impression that Erlang was (sort of) OO (If you waved your hands a lot) but not really (If you listened to what we actually said, and read the small print carefully)."

http://www.sics.se/~joe/bluetail/vol1/v1_oo.html

Erlang is closer to original Smalltalk than most of current OO languages: there are processes (objects) which communicate by messages (messages).

I also should mention OOHaskell: http://homepages.cwi.nl/~ralf/OOHaskell/ "In the second and major phase, we systematically substantiate that Haskell~98, with some common extensions, supports all the conventional OO features plus more advanced ones, including first-class lexically scoped classes, implicitly polymorphic classes, flexible multiple inheritance, safe downcasts and safe co-variant arguments. Haskell indeed can support width and depth, structural and nominal subtyping. We address the particular challenge to preserve Haskell's type inference even for objects and object-operating functions. Advanced type inference is a strength of Haskell that is worth preserving. Many of the features we get ``for free'': the type system of Haskell turns out to be a great help and a guide rather than a hindrance."

So everyone "can has" his OO…

That's because we don't actually have any concrete evidence one way or the other. This is the biggest problem with the whole static/dynamic debate.

Until there are studies that clearly demonstrate that projects written in statically typed languages have statistically less defects, are developed faster, or are easier to maintain, it's all anecdotal evidence.

There are tons of large real world projects developed in both static and dynamic languages, there's very little indication that projects developed in dynamic languages have significantly more defects or that they're more difficult to maintain.

In fact, some of the largest and most robust applications are written in Erlang. See the paper from Joe Armstrong as an example (https://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf).

Lots of people starting to use Erlang needs fault tolerance for high availability, and high concurrency because they need to scale massively.

YAWS, one of the more-popular webservers written in Erlang (that has docs, unlike Mochiweb), blows Apache out of the water when it comes to serving many users (http://www.sics.se/~joe/apachevsyaws.html). Apache can serve about 4,000 parallel sessions on the same hardware YAWS can serve at least 80,000 parallel sessions. This is due to the high level of concurrency that Erlang enables and encourages.

Another example is ejabberd, one of the most popular Jabber/XMPP servers. It takes advantage of the high concurrency also, which allows is to process many more connections than Openfire or the C-based jabberd and jabberd2.