Mailinator Blog

Fulltime, Palo Alto - funded Series A

http://www.refresh.io
paul at refresh.io

Title: "Rather Impressive Java Developer"

Our current dev team is pretty awesome and we're looking to add to it. Currently we have a few PhD's, few ex-googler's, some tech book authors, couple of ex-linkedin, ex-microsofties, and stanfords, and a bunch of startup vets like Trulia and dealer.com. Heck we even have the guy who wrote the Java testing framework TestNG ( http://www.testng.org ) and the guy who wrote Mailinator ( http://mailinator.blogspot.com ). And we're only up to 7 people. But we want an 8th and maybe a 9th.

Job Description:
Someone we want to work with. Someone who will make us smarter and better and, in return, someone we can make smarter and better too. Someone who will help us make our company and its products great.

Our technology includes:
Java. Thousands of threads (Mailinator guy keeps adding more, TestNG guy keeps removing them). Tons of data, tremendous data analysis, fuzzy logic, machine learning, natural language processing. We use Mongo and we're having no problems with it - because we're careful. We treat every superfluous database access and every unnecessary network call like its a small insult to our users. And gosh darn it, we love our users.

Simply, we are building a product that fetches and analyzes massive amounts of data to improve our user's lives (see the website for a more specific description, or better yet, email me)

Job Requirements:
You…

Mailinator has a blog, at http://mailinator.blogspot.com - you should definitely read everything on there. Some things I recall reading:

- Mailinator is designed and written from the ground up to be efficient by someone who seems pretty good (Paul Tyma; on the implementation side, it's completely custom SMTP-to-web software keeping e-mail in compressed form in memory; on the design side, it "sheds load" far more aggressively than mail servers that are intended to be reliable, necessary for the constant DoS attacks);

- Mailinator has lots of imitators, but they tend to find out that the cost of running servers is higher than the ad revenue (see "efficient and cheap", mailinator processes a ridiculous amount of e-mail and recently added its second server because getting that much bandwidth to a single server was more expensive);

- As I understand, this is a hobby for Paul; he's not exactly getting rich (well, not from Mailinator anyway; he's on LinkedIn if you want an idea.)

I don't mean to discourage you, it sounds like a nice project. Just don't expect to get rich off it, or to be done tomorrow...

Preforked and threaded blocking IO consumes far more resources than event-driven non-blocking network IO. Process/thread stacks consume a significant amount of memory. This is why nginx ruins Apache at almost anything at scale (especially reverse proxying). Apache falls over while consuming grandiose amounts of memory in the low millions of connections per day while nginx can easily handle 10mm connections per day on a single core machine with 256mb ram. It's ridiculous.

On the other hand, for long-lived connections, you're better off using blocking IO since it's more efficient. Most web traffic is not long-lived, though. See Paul's interesting article on NIO vs IO performance:

http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-is-not-faster-than.html

http://www.mailinator.com/tymaPaulMultithreaded.pdf

And don't forget about Slowloris. If you see your competition using Apache, you really don't need to worry. :)

Disclaimer: I have nothing personal against Apache and used it for six years or so. I've just moved on to better software. If it works for you, that's great.

You can't get a full list of them on mailinator.com, since it makes it easy for a misguided website administrator to just ban the whole list.

Instead, the site will randomly show one of the alternate domains on every page load. At one point, it would even give 'gmail.com' and other legitimate domains as the alternate if you tried to scrape them too quickly (or rather, they hypothetically-yet-definitely-didn't-do that).[1]

Today, the alternate domains are shown as an image[2], and even that isn't a complete list, since other people can simply redirect mail incoming to their domains to mailinator.com. Mailinator explicitly condones this.[3]

[1] http://mailinator.blogspot.ca/2011/05/how-to-get-gmailcom-banned-not-that-i.html

[2] http://www.mailinator.com/get_alt_domain

[3] http://mailinator.blogspot.ca/2007/10/new-mailinator-alternate-domains.html (end of second paragraph)

> Asynchronous, callback-based code is frequently more difficult to write, harder to reason with, more inflexible, and buggier than similar synchronous code.

That is an excellent point that is usually ignored. Thread per request is much easier to understand and work with. Of course, thread per request is simply not a good choice in cases like a server with support for web sockets, but it is good enough in many cases. My experience has mainly been with Java (NIO vs. simple threads and Executors). Paul Tyma has made a more convincing argument than me about this -

http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-is-not-faster-than.html

http://paultyma.blogspot.com/2008/03/writing-java-multithreaded-servers.html

http://www.theserverside.com/discussions/thread.tss?thread_id=26700

The best thing about mailinator other then the service is the blog posts. There is some seriously interesting hacker reading in there such as,

http://mailinator.blogspot.com.au/2012/02/how-mailinator-compresses-email-by-90.html

http://mailinator.blogspot.com.au/2011/05/notes-from-programming-leap-to.html

http://mailinator.blogspot.com.au/2010/02/how-i-sped-up-my-server-by-factor-of-6.html

http://mailinator.blogspot.com.au/2007/01/architecture-of-mailinator.html

I think most have hit the front page of HN before but its worth including here for those who have not read them.