Twitter's engineering team posts about the company's infrastructure, open-source releases, search, and deployment tools.
HN mostly links it as an example of a company engineering blog, and for the Storm stream-processing announcement.
IMO if you build a product and it is working, there really is not much reason to redo your efforts in a higher performance framework unless your needs require you to do so. If you are at your current hardware's limits and you can't throw more hardware at it (for whatever reason), it may make sense in that case. If you want to reduce the amount of servers you currently have, it also makes sense there. http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html This is a great post talking about going from 30 servers using Ruby to 2 servers using Go.
Use case example - Twitter was using Rails on the front-end for a while but it got to the point where using it made little sense. Part of Rails' performance is dependent on caching - which is really hard, if not impossible, to do well when you are dealing with live data.
FYI, Gemini is our framework that we've built and maintain internally. You probably won't find reference to it (except for out blog).
edit This is the link to Twitter's blog about Rails http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster_1656.html
Second that about Lift too. It's basically server-side jQuery. Instead of parsing templates as one big string, it does DOM transforms on sub-trees. Parallel, independent, lazy-loading transforms using an Actor for each for each Snippet/subtree. Really awesome model.
Also, Twitter's recent 180 on shifting some processing from the client back to the server [1] suggests a server-oriented framework like Lift's is worth considering, despite the hype and excitement around client-heavy Rich Web Apps. Lift does the API-consuming rich client model as well, but its uniqueness is the DOM transform stuff.
Very different mental model than MVC and hard to get started with for some because of that, but here's a great bit of advice on best practices I came across recently (especially the part about pretty everything being done via either LiftRules or the S object) [2].
1. http://engineering.twitter.com/2012/05/improving-performance-on-twittercom.html
2. http://www.quora.com/Lift-web-framework/What-are-the-best-practices-for-using-Lift
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-best-engineering-blogs
Pip and virtualenv solve the problem of creating and populating isolated runtimes for things you control. They don't really address distribution to end users, i.e., those who use OS packaging or app stores.
Distribution of python "binaries" is a real pain though. Twitter's PEX (https://engineering.twitter.com/university/videos/wtf-is-pex) is the best, in my opinion. (It's basically a packaged venv, so in that regard, I guess I agree.) It's still problematic and requires a writable filesystem just to run.
I build OS packages for python programs using PEX, and it's okay, but it's probably my least favorite distribution mechanism.
A completely stupid thing for Twitter to do, like writing a post about their 12 hours of downtime ending in rollback by
1. starting the post with 5 paragraphs and a list of all the virtues of Twitter's engineering
2. spin the fact that, after a simple DB screw-up in production it took them about 6 hours to decide to just rollback from a backup
3. end the "postmortem" (which is, by the way, completely devoid of apology) by trying to hire
all the while talking about how working at Twitter is like building a rocket mid-flight (so, just like every other company)?
http://engineering.twitter.com/2010/07/twitter-performance-update.html
Twitter solves an elementally different problem to Facebook, which may lead to the disparity in features that you see and has previously caused issues with scalability. The Twitter engineering blog (http://engineering.twitter.com/) details some of the fundamental systems development we've been doing to solve these issues and give us a better platform.
From this, we can launch better products. I can't go into specifics, but there's some really good stuff coming in the future.