Live data from Hacker News

TCP incast: What is it? How can it affect Erlang applications?

snookles.com

11–17 of 17 posts

Re: TCP incast: What is it? How can it affect Erlang applications?

#11
post #8
post #5

Computer Science includes research into networking. TCP incast is a way of describing network traffic statistical properties, particularly in data centers [1] A typical CS student's view of the network is that "everything is so random that it averages out." The exact opposite is actually the case most often, to the chagrin of your local network admin. Network traffic tends to act like everyone knew you were going to…

That's right -- bufferbloat introduces more problems than "solving" incast. One of the main reasons for incast is the synchronised bursts+backoffs causing senders to timeout. As the CMU paper pointed out, the 200ms min-RTO is too conservative for senders to recover from timeouts. Reducing it can go a long way in mitigating the incast effect. There's a project called "R2D2" at Stanford University that proposed adding…

The degenerate/extreme/unrealistic case of Incast is you have switch buffer capacity to store N segments, you talk to M servers that each return one segment, and M >> N. Although RTO is calculated based on RTT and RTTVAR, in the extreme case you can get clumps (and waves) of retransmissions (depending on the properties of the network) such that even eliminating the minRTO altogether may not solve the problem at some scale. In simulation we experimented with adding an adaptive staggering to the exponential backoff algorithm and found that it helped at high server counts [1], but it was only simulation so I'd take that approach with a grain of salt.

The R2D2 work is pretty neat: different than a lot of other approaches I've seen. I'm excited to see how the FPGA implemention works!

Some comments: 1) I think any significant change to the control algorithm requires careful analysis: the variance in throughput with the multi-client experiment looks interesting, though I don't know whether that is steady state. From the graphs, R2D2 suffers more with larger filesizes whereas TCP actually improves. 2) Real datacenters can have very different traffic patterns that can break some of the assumptions about bandwidth uniformity and latency, though it's harder for academics to tackle that. 3) If you are going down the path of TCP offload, you presumably can avoid the overhead of CPU interrupts/timer programming when reducing the RTO into microseconds :). I'd be interested in seeing how R2D2's algorithms/constants work when you're able to reduce the 3ms timer to microseconds in hardware!

Also, if some kernel programmer wants to fix my once-working patch to support microsecond-granularity TCP retransmissions [2], I personally know a bunch of people who would be happy :)

[1] http://vijay.vasu.org/static/papers/sigcomm147-vasudevan.pdf

[2] https://github.com/vrv/linux-microsecondrto

Re: TCP incast: What is it? How can it affect Erlang applications?

#12
post #5

Computer Science includes research into networking. TCP incast is a way of describing network traffic statistical properties, particularly in data centers [1] A typical CS student's view of the network is that "everything is so random that it averages out." The exact opposite is actually the case most often, to the chagrin of your local network admin. Network traffic tends to act like everyone knew you were going to…

Good luck trying to convince your virtual server hosting service provider to provide R2D2 or similar.

Re: TCP incast: What is it? How can it affect Erlang applications?

#13
post #5

Computer Science includes research into networking. TCP incast is a way of describing network traffic statistical properties, particularly in data centers [1] A typical CS student's view of the network is that "everything is so random that it averages out." The exact opposite is actually the case most often, to the chagrin of your local network admin. Network traffic tends to act like everyone knew you were going to…

What are your thoughts about flat networks (mentioned by someone in the comments on the original article)?

Re: TCP incast: What is it? How can it affect Erlang applications?

#14

Yikes, that switch is only buffering a third of a millisecond's worth of packets. Easy to see why the connections would collapse.

In many cases this is a good thing (tm). The large buffers tend to "hide" retransmits and dropped packets with a buffelength of latency, which will instead increase the burstiness pattern in many cases.

Re: TCP incast: What is it? How can it affect Erlang applications?

#16
post #9

The question might be, can we use iproute to change TCP's RTO in Linux yet?

yes.

http://www.kernel.org/doc/Documentation/networking/ip-sysctl...

look at rto_min, rto_max, rto_initial.

you can do something like:

  ip route replace dev eth0 rto_min 10ms

Re: TCP incast: What is it? How can it affect Erlang applications?

#17
post #16
post #9

The question might be, can we use iproute to change TCP's RTO in Linux yet?

yes. http://www.kernel.org/doc/Documentation/networking/ip-sysctl... look at rto_min, rto_max, rto_initial. you can do something like: ip route replace dev eth0 rto_min 10ms

Thanks!
Post reply on HN