The Bitly engineering team writes about distributed systems, monitoring, Hadoop, and open source tools like the NSQ message queue.
HN commenters point to its NSQ post for the design background of the message queue, and compare it with beanstalkd and other engineering blogs.
> Apart from Google (who uses it for Youtube), Go hasn't really been used in large projects.
I don't know how large each of these projects is, but a growing number of companies are using it for serious projects:
Iron.io: How We Went from 30 Servers to 2 with Go
http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html
Behind the Scenes: Big Data at Torbit
http://torbit.com/blog/2013/02/19/big-data-at-torbit/
pool.ntp.org DNS server in Go
http://news.ntppool.org/2012/10/new-dns-server.html
Juju at Canonical
http://www.reddit.com/r/programming/comments/18atce/juju_canonical_109k_lines_of_go_code/
Go at bitly
http://word.bitly.com/post/29550171827/go-go-gadget
CloudFlare blows hole in laws of Web physics with Go and Railgun
http://arstechnica.com/information-technology/2013/02/cloudflare-blows-hole-in-laws-of-web-physics-with-go-and-railgun/
dl.google.com now served by Go
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/BNUNbKSypE0
Go At Conformal
https://www.cyphertite.com/blog.php?/archives/7-Go-at-Conformal..html
Go at Novartis
https://plus.google.com/114945221884326152379/posts/d1SVaqkRyTL
Go at the BBC
http://www.quora.com/Go-programming-language/Is-Google-Go-ready-for-production-use/answer/Kunal-Anand
Go at SoundCloud
http://backstage.soundcloud.com/2012/07/go-at-soundcloud/
Go at CloudFlare
http://blog.cloudflare.com/go-at-cloudflare
Go at Heroku
http://blog.golang.org/2011/04/go-at-heroku.html
Building StatHat with…
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-best-engineering-blogs
I do believe marketing has something to do with it. I don't believe a random Google employee is allowed to disclose stuff about their infra permission is granted. This is not a bad thing mind you. Google is merely trying to win minds in the most friendly way possible.
Neither did I said that this was a 'major' differentiator between Go and D. Just 'a' differentiator and not a major one at that.
I have become slightly myopic with regard to the little announcements. I'm confusing it with announcements like Bitly's: http://word.bitly.com/post/29550171827/go-go-gadget or Cloudflare's: http://blog.cloudflare.com/cacheing-the-uncacheable-cloudflares-railgun-73454. But the point still stands, I see more of these posts related to Go than posts related to D (or Clojure/Scala/Rust/...).
beanstalkd is a great project and it certainly inspired some of our thinking. I think of beanstalkd as a more fully functional alternative to simplequeue (what we built/used prior to NSQ - more details in our blog post http://word.bitly.com/post/33232969144/nsq).
Some of the goals of NSQ transcended just replacing our specific daemon that buffered and delivered messages (most importantly the interactions with the lookup service). Because of that, we felt that owning that piece would make it easier to achieve those goals.
Additionally, one of the most important properties of nsqd (the queue component of NSQ) is that data is pushed to the client rather than polled (like in beanstalkd).
Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq
In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.
The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.
If there are any specific numbers you are curious about, please ask.
There are plenty accounts of Go in production: (Google's dl.google.com) https://groups.google.com/d/msg/golang-nuts/BNUNbKSypE0/E4qSfpx9qI8J, (CloudFlare's Railgun) http://blog.cloudflare.com/go-at-cloudflare, (Bit.ly) http://word.bitly.com/post/29550171827/go-go-gadget, (SoundCloud) http://backstage.soundcloud.com/2012/07/go-at-soundcloud/, (Disqus's realtime-backend) http://blog.disqus.com/post/51155103801/trying-out-this-go-thing, etc