Jim Gettys

This article is part of a series, which examine performance problems in high-speed internet connections. Unfortunately, in addition to being split over several pages, it is written in one of the most rambling and verbose styles I've ever seen. The best single explanation is found not in the submitted link, but in < http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-mastermind-bufferbloat/ >

My background in networking isn't strong enough to fully understand the problem, but here's an attempt:

--

When using a fast Internet connection, in the 20+ Mb/s range, throughput and latency for common residential use cases are both far worse than expected[1,3,4]. The problem is masked from most users because Windows XP is missing a feature required for maximum network performance, and because most American residential Internet connections are slow. As more users upgrade their operating systems and connection, significant performance will be lost.

The problem occurs due to excessive buffering, combined with the TCP window scaling feature. When a network stack with support for TCP window scaling is allowed to connect to a fast link, the link is saturated. Normally the dropped packets should signal the stack to back off, but buffering in the stack and local switch combine to cause massive latency (on the order of 10-20 seconds) before packets begin dropping. This latency, which the author terms "bufferbloat", is responsible for the performance degradation.

The author goes…

> The problem isn't that your videoconferencing goes from 1080p to 360p -- it's that it stutters and stops working for seconds at a time, so students completely miss content, and introduces a lag

Bufferbloat.

Ideally, at least for TCP applications, with perfect network congestion control, reduced bandwidth due to streaming should only degrade data rate, but never massively increase the latency. A slight increase is normal, but a huge increase is not. Unfortunately, in real life, it's often the case, and this aspect is often overlooked by vendors, developers, and sysadmins, making things even worse than it should be.

Network hardware or software is optimized for throughput, and its performance under heavy traffic is often not tested. One common practice is using a buffer that is as large as possible, so that you can push the data rate to the maximum and avoid dropping packets at all cost. This is called Bufferbloat and it's a latency disaster. When the upstream bandwidth is saturated, the downstream buffer keeps accepting more bytes/packets, effectively breaks the proper feedback signal used by congestion control algorithms, so it never kicks in in a timely manner, and the FIFO nature of the buffer means the latency of a saturated link is always as high as the buffer size. As long as something is using the bandwidth, the network will always be slow like a snail. When people start having this problem, they simply blame the bandwidth-intensive application, and set a QoS up…

Normally, TCP waits for the buffer to exceed the maximum segment size before sending any data. This creates serious delays when the two sides of the connection are exchanging short messages and need to receive the response before continuing. For example, the login sequence at the beginning of a telnet session begins with the short message "Login", and the session cannot make any progress until these five characters have been transmitted and the response has been received. This process can be seriously delayed by TCP's normal behavior when the message is provided to TCP in several send calls.

http://en.wikipedia.org/wiki/Transmission_Control_Protocol

Basically, there's a bunch of large TCP buffers between you and the rest of the internet, which can fill up when you're trying to transfer large amounts of data. When these buffers are full, important packets get queued up for long periods of time, causing latency. This is why running something like bittorrent causes your ping time to go up.

It's especially a problem when uploading, because upload speeds are often capped at a tiny fraction of your download speed, so it takes much longer to drain the buffers and get to the important packets.

See also this post:

http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-mastermind-bufferbloat/

I think you're missing the same thing the author missed in the Jim Gettys' article. You cannot fix the situation easily in every situation if the ISP is not behaving correctly, contrary to what the post says. He was able to fix this specific scenario, because he was the bottleneck (he was generating more traffic / accepting promise of delivery) in this case. The original series of articles (I really recommend reading them - they've got a great deal of tech details) dealt with a bit different situation - the person was not even saturating the link. Problems started appearing because of the shaping on a link which actually should support the traffic - he could introduce fake bottleneck, which was not the point really, because he would be wasting some possible bandwidth. The problem there was mostly out of his control, although could be mitigated to some extent by local tuning (http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-bufferbloat-in-home-routers-and-operating-systems/).

I didn't want to say your solution won't work. But it's not going to solve all problems - it will work if you / your direct link is the bottleneck.

The linked Gettys article is fantastic too:

https://gettys.wordpress.com/2010/12/17/red-in-a-different-light/

VJ is at PARC now.

The old VJ LBL papers are still collected at ICSI/ICIR (Vern Paxson is still there!):

http://www.icsi.berkeley.edu/cgi-bin/pubs/browse.pl?groupid=000002

There used to be some attention paid to deliberate evasion of congestion avoidance (Stefan Savage did a paper on it), the idea being that a malicious user could monopolize a link by ignoring congestion signals (interview question: when you upload a file via Wi-Fi over your DSL connection, how does your computer know how fast to send packets?). Maybe some of the reason that turned out not to matter was, we were in a temporary period of not needing congestion avoidance.

Finally, as I understand it, at least some of the fundamental ideas behind VJ's "hydrodynamic" congestion avoidance scheme are derived from Raj Jain's CUTE, which was done for DEC in '85.

Here's an interesting interview [1] [2] with Jim "Internet Plumber" Gettys [3], one of the original designers of X [4], and editor of the HTTP/1.1 specification. He talks about his work on the OLPC project, X memory usage, Linux and other stuff. He also has a great blog [5], and has been investigating the effect of bufferbloat on network performance [6] [7].

[1] Part 1: https://lwn.net/Articles/188060/

[2] Part 2: https://lwn.net/Articles/188073/

[3] Jim Gettys: https://en.wikipedia.org/wiki/Jim_Gettys

[4] The (Re)Architecture of the X Window System (July 2004 Ottawa Linux Symposium): https://keithp.com/~keithp/talks/xarch_ols2004/xarch-ols2004-html/

[5] Blog: https://gettys.wordpress.com/

[6] Bufferbloat: https://en.wikipedia.org/wiki/Bufferbloat

[7] Bufferbloat project: https://www.bufferbloat.net/projects/