Airbnb's engineering team writes about its infrastructure, open source tools, front-end architecture, testing and engineering culture.
HN commenters list it among good company engineering blogs, alongside Instagram and Stripe, and point to it for Airbnb's engineering work.
i am constantly impressed at the hashicorp guys, who continue to release great tools. they actually released serf on the same day as we released nerve and synapse, which comprise airbnb's service registration and discovery platform, smartstack. see https://github.com/airbnb/nerve and https://github.com/airbnb/synapse
that said, as i wrote my blog post on service discovery ( http://nerds.airbnb.com/smartstack-service-discovery-cloud/ ),
dns does not make for the greatest interface to service discovery because many apps and libraries cache DNS looksups.
an http interface might be safer, but then you have to build a connector for this into every one of your apps.
i still feel that smartstack is a better approach because it is transparent. haproxy also provides us with great introspection for what's happening in the infrastructure -- who is talking to whom. we can analyse this both in our logs via logstash and in real-time using datadog's haproxy monitoring integration, and it's been invaluable.
however, this definitely deserves a look if you're interested in, for instance, load-balancing UDP traffic
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-best-engineering-blogs
There is an entire blog dedicated to the topic of tech at Airbnb: http://nerds.airbnb.com/. Perhaps you will still feel the same after reading this?
Peter Thiel's Zero To One described four pillars that were necessary for a monopoly. It appears Airbnb has most if not all of these.
1. Network effects - this is usually more important than proprietary tech. They have this - global network effect. This is not a "retail company" any more than Amazon's 3rd party marketplace or Alibaba's is.
2. Tech - read blog and decide for yourself. It's easy to imagine at sub-scale also being able to build Facebook, Instagram, Snapchat, etc. Most of the challenges come at scale (and are protected by #1 above).
3. Brand - the commenters seem to agree they have this.
4. Economies of scale - they have this.
I think really what happens is Node enables a new type of stack[1]. It doesn't actually really step on anything new the SPA's aren't already doing. Instead it moves to become isomorphic, so initial render is done by Node, but still the data sources, validation, and heavy backend logic all stays behind an API. That API could certainly be Erlang powered.
Also to respond to your points...
1. This I see as the most valid argument, but not a deal-breaker (see Walmart, LinkedIn using Node without problems scaling).
2. Check out Flow by Facebook [2] or TypeScript.
3. Not true with require/modules.
4. It's incredibly easy to include a JSLint file in your repo and as a git hook.
[1] http://nerds.airbnb.com/isomorphic-javascript-future-web-apps/
[2] http://flowtype.org/
"The time for Dockers and containers isn't quite today, but it's very close."
Hmmm ... tell that to AirBnb (http://nerds.airbnb.com/future-app-deployment/), New Relic (https://blog.newrelic.com/2014/08/12/docker-centurion/ ) and Spotify (https://blog.docker.com/2014/06/dockercon-video-docket-at-spotify-by-rohan-singh/)
It is a young technology, but moving quickly. Just 10 years ago, YouTube wasn't owned by Google yet and we didn't have the first iPhone.
Hell, 17 years ago VMware (arguably the king in the virtual machine software market) was founded. If it was a kid, it wouldn't have graduated from high school yet!
It's called Smartstack and is also entirely open-source - http://nerds.airbnb.com/smartstack-service-discovery-cloud/
Having been a long-time user, I can tell you the added complexity of every service having to allocate its own port, then tracking that port across every other service is a lot of overhead on its own.
Not to mention you also have to run a zookeeper cluster and two additional daemons on every instance.
All in all it's a good concept, but slightly too complex for my needs. Recently made the switch back to using internal ELB's for every service and not looking back.