Dan Kegel

Take a look at the c10k problem, a classic resource for this problem domain: http://www.kegel.com/c10k.html

You probably don't want to spawn one thread per connection, but get notified by the operating system when one of a bunch of TCP socket connections has an event for you to handle. You can achieve this by the kernel calls epoll, poll, select, kqueue. For Python, I recommend taking a look at tornado (https://github.com/facebook/tornado), an asynchronous socket handling framework. In particular the "io-loop", which you could use for handling the connections.

As for serialization protocols, don't roll your own but consider JSON for maximum flexibility, and protocol buffers for maximum efficiency. For communication between servers consider well known libraries and protocols such as AMQP (eg RabbitMQ) and Redis pubsub.

Be sure to read up about Linux kernel settings to improve the number of maximum open sockets, etc. Here are several good readings referenced: http://serverfault.com/questions/10852/what-limits-the-maximum-number-of-connections-on-a-linux-server/10919#10919

Also, you'll want to get familiar with some tools you'll be using a lot, such as lsof and netcat.

About the infrastructure design: you'll most likely end up with several different server applications and possibly a number of simultaneous running instances. Consider Amazon EC2 micro instances or something similar to start with.

I'd recommend to experiment with building several prototypes before the real…

I understand your point... in terms of numbers Java does attract careerist programmers simply by critical mass. Although I would argue that language is actually now Javascript but thats another argument entirely.

The point I was making was that there is a massive and grossly unfair generalisation solely about Java developers being made here and all over HN... that they don't really love programming, or want to write the best code possible. You might feel the language precludes that but then I would beg to differ and we are back to blub allegories.

By your own figures (which I don't agree with) the top 20% of a big number is a large amount of talented programmers you have excluded from your search because you cant get past the crud. Developers who have taste, know the JVM inside out and can write tasteful, readable, lightweight (yes) and highly performant code. Code that runs everyday in the big houses such as Google, Twitter, Linkedin...

I've interviewed many Java, Python and Javascript developers and our screening process helped to weed out the very hopeless, not perfect but most.

First a scan of their resume - is it generic, do they have experience in complex projects, a github account, a blog, contribute to open source, opinionated?

Second a real programming test that can test style, consideration of performance, typical gotchas. This should be automated so you can outright reject incorrect output... not fizzbuzz.

Other people more experienced on me have commented…

Black Hat USA 2009: Realistic probabilities in modern signals intelligence[1] attempts to provide a more realistic prediction of what could be possible. The paper is structured in a similar way to how Physics and Technology for Future Presidents[2] arrives at the physical limits of remote sensing satellites.

comScore qSearch reports that US Internet search traffic to the top 5 search engines (Google, Yahoo, Bing, Ask, AOL) is on the order of 20 billion per month or 7,600 searches per second averaged across the day[3].

You'd only need a few cheap off-the-shelf servers to process and store all of that information for eternity. See [4] for example performance figures for inserting data to a PostgreSQL database on a single server (not even using fancy sharding techniques). The C10K problem[5] is also worth noting in this context.

The reason I use search traffic as an example is that it is very high value -- it doesn't just capture what people are reading or doing, it captures what they're seeking to read and do (motivations, desires). It's also a very compact data stream and requires little or no processing and analysis. DNS query traffic, registers of source and destination phone numbers, etc also provide cheap, valuable data.

Whether or not the NSA is looking at this kind of data is of little concern when compared to what the commercial world could use this data for. The NSA has a strong incentive not to use their power too widely or they will tend to lose any advantage…

I have been an OpenBSD and FreeBSD user, as well as minor contributor, since the mid-90s. All the answers here are good, and I would like to add:

* FreeBSD is on the cutting edge of performance. Because it is a total OS (as opposed to just a kernel), performance improvements are made and integrated into the binary packages and ports build options of common packages. FreeBSD was one of the first operating systems to implement kqueue, which is now available on OS X. This was part of the reason why for years FreeBSD dominated high-performance web serving. If you asked this question 5-10 years ago, the answer would simply be 'FreeBSD performs a lot better than Linux with web serving', but that is not as true today since Linux has caught up in the 2.6.x branch (with epoll). For more info see the classic: http://www.kegel.com/c10k.html

The good servers like nginx and varnish take advantage of these performance benefits that FreeBSD offers.

(there was a similar system here with the virtual memory manager, while the Linux kernel struggled with this issue ~01-02 FreeBSD was miles ahead). FreeBSD is the starting and development point for a lot of new technologies.

2. Everything is done the BSD way, from directory layout to the rc.x startup system. It is the same and uniform and integrated with the kernel and doesn't vary across releases. There is a single reference handbook, and POSIX compliance. I have always preferred the BSD way of file system layout, and that is what I have…

I googled 'epoll "O(1)"'. You're the first hit (congratulations), followed by some random person on stackoverflow, the documentation for a CPAN module, the repetition of that documentation on github and in various package repos, and some random bloggers and mailing list responders who are either just referring to the work that you have to do to find an active fd when the wait call returns, or are confused.

The man page says nothing about O(1):

http://www.kernel.org/doc/man-pages/online/pages/man4/epoll.4.html

The original epoll writeup says nothing about O(1), and shows graphs explicitly for dead connection scenarios (iow, it never claims better performance per active fd):

http://www.xmailserver.org/linux-patches/nio-improve.html

The Dan Kegel writeup that's pretty much universally cited on this issue says nothing about O(1):

http://www.kegel.com/c10k.html#nb.epoll

The link you threw up yesterday that "flat out gets it wrong" says nothing about O(1) and simply shows epoll doing exactly what it was designed to do (again, it never claims better performance per active fd), including graphs that actually do show poll outperforming epoll with small numbers of dead connections:

http://lse.sourceforge.net/epoll/index.html

Can you show any authoritative sources who claim epoll is O(1)? If not, then yes, you built a fluffy little strawman and took him down hard.

I have no problem with your work on superpoll. I'm skeptical it will help much, guessing there will be…

There are really two parts to point 2.

2.a. is about performance. And while not specifically about C++ this article covers the main idea http://www.kegel.com/c10k.html

2.b. is about avoiding deadlocks and race conditions. Threaded programming in C++ is dangerous. For years I would have said, that I could avoid deadlocks and race conditions by virtue of being a good programmer. However, I've debugged enough of my own code to know this is not always true.

If you only use threads for data driven computation, then you greatly reduce the risks. On the other hand if you have multiple threads running around accessing anything and everything in your program then you need a lot of thread locks which is tedious to implement and not well supported in C++ (vs. say the synchronized keyword in Java), does not perform well and still it is very easy to make mistakes. The point is that there is no reason to let threads run willy nilly because you can do event driven programming in C++ (libev, libevent, libuv) and most programs (or rather most parts of a program) are not computationally intensive.

edit: Here's a great resource on the topic of threading. From the SQlite FAQ "Threads are evil. Avoid them." http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf