John Ousterhout, a Stanford computer science professor, hosts papers and notes on RAMCloud, scripting languages, log-structured file systems, and startup advice.
HN links it for his classic papers, which commenters call classic, including the threads-versus-events debate, RAMCloud for in-memory databases, and log-structured file systems.
This work is really cool and is attacking an important problem in a lot of datacenter networks. It's also great that they've released the source code too. As someone who's worked on this problem, I encourage those interested in reading up more about this.
Here are a few papers on the problem in chronological order:
1. Original paper briefly talking about the "incast" TCP problem in storage environments: http://portal.acm.org/citation.cfm?id=1049998
2. Our follow up work on that problem from a few years back: Measurement paper ( http://portal.acm.org/citation.cfm?id=1364825 ) and our initial solution ( http://portal.acm.org/citation.cfm?id=1592604 ) with a microsecond retransmission Linux patch here: https://github.com/vrv/linux-microsecondrto
3. Another paper talking about Incast in Datacenter environments, focusing on a different form of the workload: http://portal.acm.org/citation.cfm?id=1592693
4. RAMCloud - a project that briefly talks about the need for low-latency transports: http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf
5. ICTCP - paper from last year that tries to solve the Incast problem using receiver advertised window algorithms: http://conferences.sigcomm.org/co-next/2010/CoNEXT_papers/13-Wu.pdf
DCTCP tries to go beyond solving the Incast problem and focuses on trying to control buffer occupancy in datacenter environments that contain both long flows and bursty flows.
I think these papers all assume lossy link layers (Ethernet), but…
> Is it acceptable/preferred to store your entire db in RAM?
If you can fit your db into RAM that's the ideal, but of course you also have to have a reliable, persistent backup. The best current compromise is probably PCI based SSD drives.
John Ousterhout (of Tcl fame) and his group at Stanford are working on a project called RAMCloud that's exploring the feasibility of storing data in RAM at all times, using disk only as backup. See his projects page http://www.stanford.edu/~ouster/cgi-bin/projects.php for more info. IMO, given the continuous improvements in RAM prices and network latencies, this type of setup for permanent storage will be the norm in data centers in a few years.
They usually share a set of features in terms of typing, extensibility, ability to be used more as glue language than real applications, interactivity
A well known paper that discusses those capabilities is the John Ousterhout's paper for the 1998 IEEE COMPUTER, "Scripting: Higher Level Programming
for the 21st Century".
http://www.stanford.edu/~ouster/cgi-bin/papers/scripting.pdf
Ah, this paper is classic... It is a response to the other (equally classic) paper "Why threads are a bad idea" [0].
In fact, threads and events are dual [1], and one can be made as efficient and the other (given a proper implementation).
[0] http://www.stanford.edu/~ouster/cgi-bin/papers/threads.pdf
[1] Lauer and Needham. “On the Duality of Operating System Structures” in 1978;
RAM is the new disk. See http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf does a nice job of looking at the practicality. See also the April 14, 2010 video at http://ee380.stanford.edu. The Stanford RAMCloud project page, http://fiz.stanford.edu:8081/display/ramcloud/Home, tracks recent activity in the area.
Sounds basically like the fundamental research on log structured file systems done back in 1992. http://www.stanford.edu/~ouster/cgi-bin/papers/lfs.pdf