Open Networking Summit
1 HN recommender · recommended in 5 months, 2015–2017
40:04 · 32K views · Jun 18, 2015
Google's Amin Vahdat gives the ONS 2015 keynote on Google's datacenter networking, including its custom hardware, software-defined networking and very high bisection bandwidth.
HN cites it as a rare public look at Google's internal network design and how it scales to many VMs.
5 comments from Hacker News
If you're interested in datacenter networking, can't miss ONS. Amin Vahdat, the guy in charge of networking at Google, has keynoted the past two years, with hardware in 2015[1] and software in 2014[2]. Mark Russinovich also spoke about Azure's SDN, but it doesn't come close to Amin's presentation. Every once in a while you'll hear some random spec from one of these companies, and it's always pretty surprising, but Amin's team has achieved 5 Petabit/s in bisectional bandwidth. It's more than surprising. [1] this video [2] [LINK] [3] [LINK]
The numbers in this are daunting, but I personally believe massively multi-core systems make the problem a lot less daunting than the article makes out. Core counts in big servers can get up over 100 per server for Intel (see Amazon's new EC2 offerings for public evidence of this). Intel's Xeon Phi series of processors offer core counts approaching ~300. Going to 300x takes the required latencies per request from microseconds up to the millisecond range. POWER systems can go even higher. Moreover, for many workloads that actually leverage this sort of compute you can do something horrifying with the new DRAM-addressable persistent storage: DMA directly from the NIC into block storage. Some (many?) high performance network adapters offer the ability to filter packets to distinct Rx queues; buffers can be posted with addresses in the storage mapped region allowing direct NIC->storage transfer. If you bake more intelligence into the NIC, you can even do things like Mellanox's NVMe Fabrics: [LINK] This is particularly relevant to the JBOD example. Now, there's the question of what you're actually going to do with all of that data, but in a lot of cases it's likely a durable read-mostly cache that's effectively a materialized view optimized of some (hopefully much slower write-rate) transactional store (say, product data on Amazon -- detail pages served up at some absurdly high rate, but a relatively low mutation rate). Other workloads I can think of fall into a category I tend to
For some reason they didn't actually link to the talk, which I haven't watched, but presumably actually tries to start answering those questions. Quick search is showing the 2015 keynote that Amin gave, haven't found the 2017 one yet... [1] - 2015 ONS Keynote this video
They switched to openflow on internal networks for this reason, deterministic: this video External network is BGP though and sounds like they didn't detect it until user reports hit. They can't predict the problem and there detection isn't working well either.
Hi Joe :) Conveniently, Amin Vahdat recently shared some content publicly about why it should scale to lots of VMs: this video (And, to others, disclaimer: I work on some bits of GCE networking at Google)