TCP incast: What is it? How can it affect Erlang applications?
1–10 of 17 posts
Re: TCP incast: What is it? How can it affect Erlang applications?
#2> ... head-of-line blocking ... What’s the remedy?
I don't know anything about Erlang or the software they're dealing with, but I'm surprised that "program your software to avoid unnecessary head-of-line blocking" didn't make the list.
Re: TCP incast: What is it? How can it affect Erlang applications?
#3> ... head-of-line blocking ... What’s the remedy? I don't know anything about Erlang or the software they're dealing with, but I'm surprised that "program your software to avoid unnecessary head-of-line blocking" didn't make the list.
It's in the post. Cmd + F "backpressure/feedback mechanism built in to Erlang"
Re: TCP incast: What is it? How can it affect Erlang applications?
#4> ... head-of-line blocking ... What’s the remedy? I don't know anything about Erlang or the software they're dealing with, but I'm surprised that "program your software to avoid unnecessary head-of-line blocking" didn't make the list.
It's in the post. Cmd + F "backpressure/feedback mechanism built in to Erlang"
Yeah, I get that it may be non-trivial to avoid head-of-line blocking in that context, but I doubt that it's impossible. If it is impossible, maybe Erlang just isn't the best tool for the job.
From what I've read, I think you could create a separate process dedicated to sending stuff to B, and have other processes deal with other clients. It's probably not possible, but it would also be nice if you could detect these hung processes and start more workers to deal with other clients, each dealing with potentially more then one client. (yes, I may be totally wrong)
Re: TCP incast: What is it? How can it affect Erlang applications?
#5Computer Science includes research into networking. TCP incast is a way of describing network traffic statistical properties, particularly in data centers [1]
A typical CS student's view of the network is that "everything is so random that it averages out." The exact opposite is actually the case most often, to the chagrin of your local network admin.
Network traffic tends to act like everyone knew you were going to youtube right now, and everybody jumped on the network all at once. The statistical term is "self-similarity" and the Hurst Parameter (H) measures how badly a network's traffic is _not_ averaging out, but bursting as if everyone knew you were going to youtube.
It might help to mention another place we see self-similarity: fractals.
This article just breaks down the situation in a cluster. Again, self-similar traffic patterns mean everyone tries to talk at once, and their TCP stacks all back off randomly, so the total bandwidth of the network is poor.
Unfortunately, the blog post says, "What’s the remedy? We don’t have a good remedy for this yet."
Sure we do. Please google some of the relevant terms for great articles on network traffic analysis and optimization. For example "self similar network traffic" and "hurst parameter." Even the CMU site linked from the blog post has a great writeup under the section "SOLUTIONS" :) [2]
Additionally, as a comment on the blog points out, larger buffers on routers can be a really _bad_ thing! Buffer bloat tends to hide core issues with larger latencies, but not solve them.
Re: TCP incast: What is it? How can it affect Erlang applications?
#6Yikes, that switch is only buffering a third of a millisecond's worth of packets. Easy to see why the connections would collapse.
Re: TCP incast: What is it? How can it affect Erlang applications?
#7Yikes, that switch is only buffering a third of a millisecond's worth of packets. Easy to see why the connections would collapse.
Yeah. Cisco 3750s are notorious for having small buffers. They are more appropriately used for connecting desktops in a office.
Cisco 4948s are better positioned for top of rack and are more comparable to the Juniper EX4200s that were shown to be better in the Tolly Report shown in the post.
Re: TCP incast: What is it? How can it affect Erlang applications?
#8Computer Science includes research into networking. TCP incast is a way of describing network traffic statistical properties, particularly in data centers [1] A typical CS student's view of the network is that "everything is so random that it averages out." The exact opposite is actually the case most often, to the chagrin of your local network admin. Network traffic tends to act like everyone knew you were going to…
That's right -- bufferbloat introduces more problems than "solving" incast.
One of the main reasons for incast is the synchronised bursts+backoffs causing senders to timeout. As the CMU paper pointed out, the 200ms min-RTO is too conservative for senders to recover from timeouts. Reducing it can go a long way in mitigating the incast effect.
There's a project called "R2D2" at Stanford University that proposed adding a tiny shim layer that rapidly retransmits lost TCP segments. It was done in a transparent manner to hide packet losses from TCP thus preventing TCP from experiencing a timeout. You can read more about it here: http://sedcl.stanford.edu/files/r2d2.pdf. EDIT: The highlight (relevant to the blog post) is that it is a loadable kernel module requiring no changes to TCP stack!
Disclaimer: I work in the same group, and I am familiar with R2D2.
Re: TCP incast: What is it? How can it affect Erlang applications?
#9The question might be, can we use iproute to change TCP's RTO in Linux yet?
Re: TCP incast: What is it? How can it affect Erlang applications?
#10Nice article, with good related references-an excellent read-Didn't know about Tcp incast before- good way to start a Saturday morning!