Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Boosting upload speed and improving Windows' TCP stack

dropbox.tech

Post reply on HN
Print view View original

111–120 of 127 posts

Re: Boosting upload speed and improving Windows' TCP stack

#111

Post by brohee »

Agingcoder wrote: post #69

Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduli…

I think the problem disappearing while running tcpdump is one of the truest instances of Schrödingbug...

Re: Boosting upload speed and improving Windows' TCP stack

#112

Post by Agingcoder »

brohee wrote: post #111
Agingcoder wrote: post #69

Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduli…

I think the problem disappearing while running tcpdump is one of the truest instances of Schrödingbug...

It was perfectly reproducible.

The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on).

I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.

Re: Boosting upload speed and improving Windows' TCP stack

#113

Post by baruch »

Agingcoder wrote: post #69

Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduli…

Networking actually has tons of interesting and complex math. Congestion control is a rabbit hole of math and control theory.

Re: Boosting upload speed and improving Windows' TCP stack

#114

Post by SaveTheRbtz »

slowstart wrote: post #106
SaveTheRbtz wrote: post #101

Earlier quoted context omitted.

A couple of questions: * What are the reasons for disabling TCP timestamps by default? (If you can answer) will they be eventually enabled by default? (The reason I'm asking is that Linux uses TS field as storage for syncookies, and without it will drop WScale and SACK options greatly degrading Windows TCP perf in case of a synflood.[1]) * I've noticed "Pacing Profile : off" in the `netsh interface tcp show global` o…

Windows historically defaulted to accepting timestamps when negotiated by the peer but didn't initiate the negotiation. There are benefits to timestamps and one downside (12 bytes overhead per packet). Re. syncookies, that's an interesting problem but under a severe syn attack, degraded performance is not going to be the biggest worry for the server. We might turn them on but for the other benefits, no committed plan…

re. syncookies: Linux by default starts issuing syncookies when listening socket's backlog overflows, so it may be accidentally triggered even by a small connection spike. (This, of course, is not an excuse for a service misconfiguration but it is quite common: somaxconn on Linux before 5.4 used to be 128 and many services use the default.)

re: pacing: Awesome!! I would guess it is similar to Linux "internal implementation for pacing"[1]. Looking forward to it eventually graduating form being experimental! As a datapoint: enabling pacing on our Edge hosts (circa 2017) resulted in ~17% reduction in packet loss (w/ CUBIC) and even fully eliminated queue drops on our shallow-buffered routers. There were a couple of roadbumps (e.g. "tcp: do not pace pure ack packets"[2]) but Eric Dumazet fixed all of them very quickly.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re: Boosting upload speed and improving Windows' TCP stack

#115

Post by the8472 »

slowstart wrote: post #97
the8472 wrote: post #89

Earlier quoted context omitted.

Are equivalents to linux' BQL/AQL, fq_codel, TCP_NOTSENT_LOWAT in the pipeline?

I cannot comment on queuing disciplines and limits in future products. Re. TCP_NOTSENT_LOWAT, you may want to look at the Ideal Send Backlog API that allows an application to have just more than BDP queued to keep the performance at the maximum throughput while minimizing the amount of data queued: https://docs.microsoft.com/en-us/windows/win32/winsock/sio-i...

Interesting. ISB does seem less ergonomic for blocking IO but it makes sense for overlapped requests.

Re: Boosting upload speed and improving Windows' TCP stack

#116

Post by stephc_int13 »

michaelmcmillan wrote: post #47
stephc_int13 wrote: post #16

Is TCP the best choice? Why not UDP?

And reimplement TCP on top? Would not recommend.

Why not?

I am not saying that it should be done from scratch. But most recent research done in the recent years about the protocols used on the web tend to be built on top of UDP and not TCP, for many historical reasons.

In theory TCP would be the better choice, but in practice this is more complex than you assume.

I think that many people have a knee-jerk reaction when talking about TCP vs UDP, but they probably don't know as much as they think... (parrots)

Re: Boosting upload speed and improving Windows' TCP stack

#117

Post by thrdbndndn »

alternatetwo wrote: post #108
thrdbndndn wrote: post #78

Earlier quoted context omitted.

Tried to change upload speed to no limit, doesn't make much difference. Ping result: Pinging nsf-env-1.dropbox-dns.com [162.125.3.12] with 32 bytes of data: Reply from 162.125.3.12: bytes=32 time=27ms TTL=55 Reply from 162.125.3.12: bytes=32 time=27ms TTL=55 Reply from 162.125.3.12: bytes=32 time=27ms TTL=55 Reply from 162.125.3.12: bytes=32 time=27ms TTL=55 Ping statistics for 162.125.3.12: Packets: Sent = 4, Receiv…

Yeah, Windows 7's speeds with single connections are very, very slow. It took me years to figure out the reason were bad TCP settings, because it worked fine when I used a multiconnection downloader. What fixed it for me was running these 3 commands in a CMD with admin permissions: netsh interface tcp set heuristics disabled netsh int tcp set global autotuninglevel=normal netsh int tcp set global congestionprovider=c…

Thanks for the tips.

I tried, and the first two lines helped the speed bump to 2.5MB/s! The third one doesn't seem to have any immediate effect.

Still not OneDrive level, but I'm more than happy.

Re: Boosting upload speed and improving Windows' TCP stack

#118

Post by Natfan »

SaveTheRbtz wrote: post #109
Agingcoder wrote: post #69

Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduli…

> tcpdump ... linux TCP stack ... ebpf > Networks are not my specialty I wish all network non-specialist were like you!

I have nowhere near any of these skills, and I know I'm not a specialist.

Re: Boosting upload speed and improving Windows' TCP stack

#119

Post by michaelmcmillan »

stephc_int13 wrote: post #116
michaelmcmillan wrote: post #47

Earlier quoted context omitted.

And reimplement TCP on top? Would not recommend.

Why not? I am not saying that it should be done from scratch. But most recent research done in the recent years about the protocols used on the web tend to be built on top of UDP and not TCP, for many historical reasons. In theory TCP would be the better choice, but in practice this is more complex than you assume. I think that many people have a knee-jerk reaction when talking about TCP vs UDP, but they probably don…

The onus is on you to explain why. Why not: Smaller payloads per packet and missing out on all the TCP algorithms already implemented in hardware en route.

Re: Boosting upload speed and improving Windows' TCP stack

#120

Post by slowstart »

SaveTheRbtz wrote: post #114
slowstart wrote: post #106

Earlier quoted context omitted.

Windows historically defaulted to accepting timestamps when negotiated by the peer but didn't initiate the negotiation. There are benefits to timestamps and one downside (12 bytes overhead per packet). Re. syncookies, that's an interesting problem but under a severe syn attack, degraded performance is not going to be the biggest worry for the server. We might turn them on but for the other benefits, no committed plan…

re. syncookies: Linux by default starts issuing syncookies when listening socket's backlog overflows, so it may be accidentally triggered even by a small connection spike. (This, of course, is not an excuse for a service misconfiguration but it is quite common: somaxconn on Linux before 5.4 used to be 128 and many services use the default.) re: pacing: Awesome!! I would guess it is similar to Linux "internal implemen…

Thanks for the heads up. We will investigate to see what fraction of connections end up losing these options. Pacing TCP is certainly on our roadmap. Our QUIC implementation MsQuic paces by default already.
Post reply on HN