Google's engineering tools team describes its internal build system, testing, source code access, and bug prediction.
HN commenters list it among company engineering blogs and cite it for Google's build and bug-prediction practices and monorepo tradeoffs.
I program in C++ at Google. I won't make claims about "stellar". Here are a few Google-specific reasons Go hasn't replaced C++ for me:
* The long C++ build times he was complaining about are mostly gone due to distributed builds. http://google-engtools.blogspot.com/2011/09/build-in-cloud-distributing-build-steps.html Throwing resources at the problem isn't ideal, but it does work.
* Google has a lot of C++ library code, including client libraries for things like Bigtable. Go's Google-specific libraries are a work in progress.
* For serving systems, I care more about machine efficiency and latency for than about development time. I'm still skeptical of those properties in Go. Obviously the compiler (and my comfort with it) will improve over time. So will the garbage collector, to a point. It will always operate on one large heap like Java's. People say Java's GC works fine, but they spend a lot of time tuning it and then quote pause times that don't thrill me. I've seen a lot of problems caused by bad tuning parameters, like (for example) Java heap sizes that exceed the size of the container it's running in. And relatively hard-to-diagnose and slow-to-recover GC death spirals because the defaults suck: Java just bogs down in situations where a C++ binary would dump a heap sample, crash, and get restarted. So I'd be more comfortable with a system like Rust's: many smaller heaps, language enforcement of ownership rules when transferring between its equivalent of…
Pointing at a big company and saying "they're doing it wrong" is easy enough to do, but you have to remember that every decision comes with tradeoffs. Take Google's codebase, since it's the one I know the best. A couple of the key decisions:
* Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication.
* Build from head. We build everything from source, statically linked. No need to worry about multiple versions of dependencies, no lag between a bug fix and it being available to any and all binaries that need it, whenever they're next updated.
I don't think that an "internal Github" is going be a magic bullet here. It's more likely it would be a matter of trading one set of hard problems for another, as we all of a sudden need to figure out how to do cross-dependencies sanely, deal with multiple versions of libraries, etc, at scale. You are correct that one monolithic Perforce repo is a bit of a pain point, but that doesn't necessarily mean that the right decision is to shatter our codebase into different pieces - we'd rather make our repo scale better. For reference, we've already got hundreds of millions of lines of code, 20+ changes/minute 6 months ago (so what, 30+ now?), and plans for scaling the next 10x are in motion.
If you're interested, I recommend http://google-engtools.blogspot.com/. It details a number of the problems we've run into, and our solutions for dealing with them at scale.
I'm not a grandmaster by any means, but these are things that I focus on:
1. Aggressively avoid techniques and abstractions that have produced bugs in the past. The "backwards conditional" is an example of this, in languages like C that will accept x = 0 as a conditional, it is safer to do 0 == x in an if statement in case you mistype. Mutability trips me up sometimes, so I try to never re-assign a variable. I keep links to the source code of key libraries in my bookmarks bar so I never have the guess how something actually works. Static analysis tools, and bug prediction tools [1] are two steps I haven't taken.
2. Seek out code written by experts and inspect it [2]. If I'm considering using a new language or library one of the first things I do is to search for large scale OSS projects. For a new language looking at some of the popular 3rd party libraries can be really illuminating. Especially if you're already familiar with the problem space.
3. two words: learn to fuckin' type.
4. Get familiar with what the history researchers call the Primary Sources. Not just the actual source code but the proposals, the specifications, and the research reports that are written by the people who conceived the tools that you are using.
[1] http://google-engtools.blogspot.ca/2011/12/bug-prediction-at-google.html
[2] (EDIT) According to Paul Allen, in high school Bill Gates actually went dumpster diving for source code:…
These are some of the top sites' engineering blogs which I collected over a period of time - I removed those which have already been posted:
Google: http://googledevelopers.blogspot.com/ http://google-engtools.blogspot.com/ http://googleresearch.blogspot.com/
YouTube: http://youtube-eng.blogspot.com/
Baidu: http://usa.baidu.com/category/baidutechblog/
Yahoo: http://yahooeng.tumblr.com/ http://yahoodevelopers.tumblr.com/
Wikipedia (closest I could find): https://blog.wikimedia.org/c/technology/
Amazon: https://aws.amazon.com/blogs/aws/
Twitter: https://engineering.twitter.com/
eBay: http://www.ebaytechblog.com/
Microsoft: http://microsoftengineering.com
FourSquare: http://engineering.foursquare.com/
Apple (closest): https://developer.apple.com/
Paypal: https://www.paypal-engineering.com/ https://devblog.paypal.com/
Dropbox: https://blogs.dropbox.com/tech/
Edit: More (collected from this link: http://www.quora.com/What-are-the-top-startup-engineering-blogs):
Yelp: http://engineeringblog.yelp.com/
Quora: http://engineering.quora.com/
Yammer: http://eng.yammer.com/blog/
AirBnB: http://nerds.airbnb.com/
Square: https://corner.squareup.com/
Flickr: http://code.flickr.net/
Soundcloud: https://developers.soundcloud.com/blog/
Cloudera: http://blog.cloudera.com/blog/
Slideshare: http://engineering.slideshare.net/
Heroku: http://engineering.heroku.com/
Pinterest: http://engineering.pinterest.com/
I believe this is Amin's paper you refer to: https://www.usenix.org/legacy/publications/library/proceedings/usenix98/full_papers/vahdat/vahdat.pdf
And for shorter reading, there's also djb's 'redo' thing: http://cr.yp.to/redo.html - specifically, the section of "Target files depend on nonexistent files" is an interesting one if you have tools that are allowed to generate dependencies for source files that aren't part of the source files.
(It turns out, if you follow that road, that your build tools and all their libraries are implicit dependencies in the graph as well. Second law of build graphs: You will at some point question if it should be legal to do unspeakable things to people who rely on pseudo-randomness in their build tools)
It is a conceptually simple problem which turns out to be surprisingly hairy in practice. That's why it's fun, and so many folks noodle on it :)
If you're interested in the field, I know Google has talked about theirs: https://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html - and I'm sure many other companies as well.
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-best-engineering-blogs