Level 3 Blog

Are you talking about this article? http://blog.level3.com/global-connectivity/observations-internet-middleman

> A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.

> That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.

So I'm reading it with my republican hat on, and I'm thinking: why is this any business of the government's? This sounds like a contractual dispute between two network companies over upgrades. Why should the government get involved in this?

The "consumer satisfaction" angle at the end might be more compelling, but more to a Ralph Nader type than even a Bill Clinton type much less a George Bush type. And the Ralph Nader types raised these objections in 1996 in opposition to telecoms deregulation, and they lost over a bi-partisan effort.

You can raise the specter…

> it's a 20:1 disparity in traffic in vs traffic out.

Are you talking about on this graph?

http://blog.level3.com/wp-content/uploads/2014/05/route_info_1.jpg

Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.

> ...transit providers are starting to see their industry be squeezed by the big ISPs.

No.

> ...why do transit providers even exist?

Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).

Take another look at the L3 network map from the blog post:

http://blog.level3.com/wp-content/uploads/2014/05/network_map.jpg

If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.

> Settlement-free peering assumes that bandwidth usage is roughly symmetric

No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.

Comcast empirically does not care…

"From the ISPs perspective, Netflix's needs are diverging from a typical free peering arrangement."

I would argue that point. I pay for Internet access, not "All of the Internet MINUS Netflix" access. As a result my ISP SHOULD (doesn't mean they will) ensure that all it's peering connections are of sufficient capacity to provide me with whatever bandwidth I choose to use up until I run into the rate limits for the plan I'm on.

My ISP sells me 25/5 I think so the idea that I can't get 3Mbps worth of Netflix is suspect.

Of course you have to consider that when you're not saturating your last-mile connection it could be a bottleneck anywhere. If it's 15 hops from me to a server if any of those 15 hops is saturated, that's the limiting factor no matter how much capacity there is before or after. That's why closing a single lane on a freeway for as little as 10 feet can really screw up traffic. You have to merge down and deal with reduced capacity even though it's not miles and miles of lane closure.

What we've been told via one of Netflix's backbone providers (Level 3) is that Netflix isn't the slowdown, the backbone provider isn't the slowdown, but the ISPs are the slowdown. How can they say this with certainty? Because they're peered with various ISPs, the connections are saturated, and the ISPs refuse to upgrade their side of the peering connection unless they're paid.

http://blog.level3.com/global-connectivity/observations-internet-middleman/

That might sound…

You might like to read Level3's take on it.

A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.

That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.

Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.

http://blog.level3.com/global-connectivity/observations-internet-middleman/

It's not just Netflix, it's also YouTube (http://www.google.com/get/videoqualityreport/), and Level3 (http://blog.level3.com/global-connectivity/observations-internet-middleman/) and you can be sure those aren't the exception.

Instead of fulfilling their obligations to provide advertised bandwidth to their paying customers (who only pay them in order to access the content provided by someone else), the last-mile providers are using their customers as hostages while they negotiate ransom from the content providers. While they also report record profits (sending their customer's payments back to stockholders instead of into the network), because the economics of their business are compelling on both the consumer side (low expectations, minimal support costs, essentially no competition) and on the technology side (network technologies increase capacity-per-$ as fast or faster than traffic grows).

Oh, also, 99% of the last-mile providers operate a sanctioned monopoly that has let them build and grow their networks with essentially guaranteed income from rate-payers and essentially no competition. Hence adding Internet is relatively low-cost and highly-profitable, but apparently not profitable enough.

I'm fairly sure that the "4.2.2.x was never meant to be public" line is a myth. Though the NANOG thread cited in that post is good historical background, it is contradicted by more modern sources:

"[...] DNS infrastructure is largely split into two types; open (public) and closed (private). Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."

- Mark Taylor, VP at Level3

http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and-dns/

I can't find any cite where anyone else who I would consider a reliable source in the DNS world (Vixie, &c) repeat this claim. To the contrary, Level 3 is often grouped with Google, OpenDNS and others in discussions of open public resolvers [1][2][3], and those in the know never seem to speak up and say otherwise in these discussions.

That being said, I have absolutely no personal knowledge on any of this.

[1] http://www.maawg.org/system/files/Fergie_DNS_Open_Resolver_MAAWG_India_SANOG.pdf

[2] http://markmail.org/message/gh7f2wvfbn5mpvuq

[3] http://www.circleid.com/posts/87143_dns_not_a_guessing_game/#4234