Google Workspace
5 HN mentions · mentioned in 4 months, 2017–2023
5:28 · 23M views · Dec 17, 2014
Joe Kava, VP of Google's Data Center Operations, gives a short tour of a Google data center. He covers security, sustainability and the core infrastructure architecture.
Cited as a documentary look at how Google runs its data centers, though sometimes dismissed as PR.
5 comments from Hacker News
50 SRE's sounds like they could keep borg running. There really isn't that much details on what borg is made up of outside of a 2015 research paper[1]. Just to address the parts you laid out. 1. Keep Borg running What would entail running? There are plenty of small teams that manage hundreds of servers and virtual machines through monitoring, deployment, networking, decom, etc lifecycle. This seems counter to purpose of cloud computing, commodity hardware and abstracted compute resources that can be deployed on-demand. Not to mention redundant sites, availability zones, <insert your cloud providers name for High Availability(HA) feature here>. Is this the part that SRE's maintain? 2. Machine maintenance From documentaries[2] and other comparable data center operations this would seem to be handled by on-site datacenter staff. Disk replacement, physical replacement, network cabling, etc. Without any other operational info I would doubt resources would just be dedicated to borg as its a bit of a overall technician site work. As for OS and software updates and again without any other info or insight would seem to fairly automated after passing testing or at least passed to another team that just handles updates. 3. Machine procurement Again commodity hardware or if its all custom still that would seem to be more of a EE task for build out and accounting for purchasing. Otherwise at Google's scale you would just get pre-populated racks delivered and replace the entire rack when 5
I appreciate the examples but none of these items are difficult regardless of size of the organization. These are standard system and network administrative items or on-site technician work. Unless you have other data about the borg architecture to show, its entire premise from the paper[1] is that it just runs jobs and removes the developers needing to know about the underlying hardware or OS that its running on it. From the paper I referenced its huge cluster and if its their own software I'm sure with the development experience at google they have accounted for loosing a cluster node. Losing any number of cluster nodes is the same scenario if you were performing an software or hardware upgrade in which case you would be taking a node offline. I would be very skeptical that there are people manually kicking off upgrade jobs for nodes in the cluster. Maybe an A/B deployment or burn-in for a week before fully pushing it out with a pipeline workflow. I'd more so say that their borg implementation probably runs at different minor versions frequently. Google runs their own data centers, thats the video documentary I referenced[2]. For machine procurement I don't see that as a SRE task. If they are making custom boards and systems they are going to have an entire group supporting that. At the size of google I'm thinking they wouldn't be running into supplier issues that they could figure out easily or account for in their borg software. Again its supposed to be one big abstractio
Here's my bigger-picture explanation of why, expressed as a set of consecutive counterpoints: 1. I initially found it curious that [LINK] ("Cloudflare proxies are dumping uninitialized memory") never seemed to made the mainstream news, while Heartbleed did. There are some awkward conspiracy arguments there, but I eventually realized that there's also the fact that people can just make mistakes at the end of the day, and while Heartbleed was very arguably in the public interest because individuals everywhere needed to take action to keep their systems (arguably) secure, making a website and silly name to highlight the security implications of a single company's mistake just... has the wrong tone, and it doesn't seem too much of a stretch to see major news coverage as somewhat similarly interfering and unhelpful. 2. There was discussion here ~some years ago about a random analytics test portal someone found that, when logged into (with "demo"/"password" or something similar) from a cellular device's IP, doxxed the name/address/last-4 of SSN (or possibly the whole thing, unsure)/etc of the account owner if that device was signed up to a particular US telco. Made quite a splash here ; never hit the news. Not only was this not a mistake, it was definitely in the public interest: the company in question was clearly buying a realtime feed of $telco's entire IP address table, an item that should simply not have been for sale, and lack of security on the purchaser's part meant an unbo
> In fact, I'd be surprised if they hadn't granted themselves some special capabilities with regards to the content of the communications. Touting their end-to-end encryption has enticed many people to trust the product, sometimes with strong conviction, while giving themselves a monopoly on access to communication perceived as secure by the end users. It reminds me of this PR puff piece[1] by Google, banging on about how secure their data centre was, the limited access by employees, the amazing information security team, the underfloor lasers to detect intruders, etc. while totally ignoring the elephant in the room, i.e. NSA backdoors which Google is forced to comply with and can't reveal publicly when they do so. [1] this video
I guess you thought wrong. What do you think this hardware tech is doing in this video? this video Men with carts scale perfectly. One guy can manage 1000 machines. 2 guys can manage 2000. Perfect scaling.