Boosting upload speed and improving Windows' TCP stack • Hacker News Board

Re: Boosting upload speed and improving Windows' TCP stack

#121

Post by drummer » Wed, May 19, 2021, 4:56 PM UTC

slowstart wrote: post #120

SaveTheRbtz wrote: post #114

Earlier quoted context omitted.

re. syncookies: Linux by default starts issuing syncookies when listening socket's backlog overflows, so it may be accidentally triggered even by a small connection spike. (This, of course, is not an excuse for a service misconfiguration but it is quite common: somaxconn on Linux before 5.4 used to be 128 and many services use the default.) re: pacing: Awesome!! I would guess it is similar to Linux "internal implemen…

Thanks for the heads up. We will investigate to see what fraction of connections end up losing these options. Pacing TCP is certainly on our roadmap. Our QUIC implementation MsQuic paces by default already.

Do you have any details on how or when Microsoft will roll out quick in Windows? Will it work by just specifying the quick protocol when creating a socket like with tcp?

Re: Boosting upload speed and improving Windows' TCP stack

#122

Post by toast0 » Wed, May 19, 2021, 5:13 PM UTC

drewg123 wrote: post #44

toast0 wrote: post #27

Earlier quoted context omitted.

I did some work optimizing a similar problem, but simpler and on another OS[1]. The basic concept that worked was Receive Side Scaling (RSS), which was developed by Microsoft, for Windows Server. Did you come accross that? It needs support in the NIC and the driver, but intel gigE cards do it, so you don't need the really fancy cards. I don't know what the interface is like for Windows, but inbound RSS for FreeBSD is…

Do you actually use RSS via options RSS / options PCBGROUP? I've tried it several times, and its just so hard to get right & have matching cores / rx rings, etc. I've made it work with a local patch to nginx, but it was so fragile that I abandoned it. I had been thinking that RSS/PCBGROUP was totally abandoned and could potentially be removed.

On further reflection, I just want to emphasize how much of an improvement RSS/PCBGROUP and the couple of minor tweaks made for this use case; with unmodified FreeBSD 12 and HAProxy and the load we had, there was basically zero conncurrency available, you could run as many processes (or threads) as you wanted, and the capacity would be the same, and it was sad, I think we could only get about 100k clients on a box before it would run out of steam.

With everything tweaked, we got to 2M clients per server, and actually it was hard to find the limit, because I wasn't able to direct enough traffic to the machines under test.

The software and configuration changes weren't big, but it was a huge impact. On the other hand, if RSS and PCBGROUP weren't in the kernel, I don't think I would have been able to add something similar, and we would have had to something wild and crazy (or try Linux and see if it would do the job). Now, I really did want to write a raw packet tcp proxy in userspace, but I knew it would be a lot easier to manage and quicker to get working with something off the shelf.

Of course, maybe there's a better solution to the root bottleneck, which was always opening a new outgoing tcp connection; even with all the tweaks, that was still the bottleneck, but fixing that needs someone more skilled than me, and I guess it's a pretty niche use case to be opening so many outgoing sockets. Accepting tons of sockets is way more common and way more optimized.

Re: Boosting upload speed and improving Windows' TCP stack

#123

Post by Agingcoder » Wed, May 19, 2021, 5:16 PM UTC

baruch wrote: post #113

Agingcoder wrote: post #69

Excellent article ! I got hit by the exact same issue which is described in the fermilab paper, namely packet reordering caused by intel drivers. It took me several days to diagnose the problem. Interestingly enough, the problem virtually disappeared when running tcpdump, which, after a lot of reading on the innards of the linux TCP stack, and prodding with ebpf, eventually led me to conjecture that it was a scheduli…

Networking actually has tons of interesting and complex math. Congestion control is a rabbit hole of math and control theory.

Indeed.

I ended up reading quite a bit about congestion control while investigating a different issue (sending data from a 25gb box to a 1gb one over a 10gb/100ms latency link didn't work well since the bigger nic would saturate the smaller one which then dropped packets, and this caused the tcp window to shrink significantly ), and it was extremely interesting. The whole problem of having multiple agents competing with different strategies and incomplete information to maximize their network throughput also reminded me of economics.

Tcp pacing essentially solved the problem (and if not available brutally traffic shape).

Re: Boosting upload speed and improving Windows' TCP stack

#124

Post by fletchowns » Wed, May 19, 2021, 8:56 PM UTC

encryptluks2 wrote: post #107

fletchowns wrote: post #98

Earlier quoted context omitted.

The files were still in Google Drive under "My Computer". My attempt was the latter, reinstalling the Backup and Sync client.

I believe simply installing the Google Drive for Desktop client would have been what you wanted to do. As the name suggests, the Backup and Sync client is well primarily for backing up and syncing your data automatically to Google Drive and Google Photos.

Thanks for the tip! I'll check that out

Re: Boosting upload speed and improving Windows' TCP stack

#125

Post by alternatetwo » Thu, May 20, 2021, 2:32 AM UTC

thrdbndndn wrote: post #117

alternatetwo wrote: post #108

Earlier quoted context omitted.

Yeah, Windows 7's speeds with single connections are very, very slow. It took me years to figure out the reason were bad TCP settings, because it worked fine when I used a multiconnection downloader. What fixed it for me was running these 3 commands in a CMD with admin permissions: netsh interface tcp set heuristics disabled netsh int tcp set global autotuninglevel=normal netsh int tcp set global congestionprovider=c…

Thanks for the tips. I tried, and the first two lines helped the speed bump to 2.5MB/s! The third one doesn't seem to have any immediate effect. Still not OneDrive level, but I'm more than happy.

Good to know which actually take any effect, since it's been so long since I cobbled those commands together that I didn't really remember either!

I probably took them from Win7 forums or some stackoverflow spinoff, the TCP problems on Win7 are not entirely unknown.

Re: Boosting upload speed and improving Windows' TCP stack

#126

Post by o-__-o » Thu, May 20, 2021, 8:10 AM UTC

Agingcoder wrote: post #112

brohee wrote: post #111

Earlier quoted context omitted.

I think the problem disappearing while running tcpdump is one of the truest instances of Schrödingbug...

It was perfectly reproducible. The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on). I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.

Pcap captures are not multithreaded so you are pinning to a single core.

This entire thread is interesting because it is highlighting a similar problem with my virtualized router. When I pinned the router vm to specific cpus the problem goes away. I switched to openstack which doesn’t give me the best control over cpu capabilities and the problem has manifested in a worse form.

My uninformed opinion is that there are underlying concurrency problems with multithreaded user land-kernel interaction and some nic drivers (consumer intel and Broadcom hardware)

Re: Boosting upload speed and improving Windows' TCP stack

#127

Post by Agingcoder » Thu, May 20, 2021, 8:38 AM UTC

o-__-o wrote: post #126

Agingcoder wrote: post #112

Earlier quoted context omitted.

It was perfectly reproducible. The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on). I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.

Pcap captures are not multithreaded so you are pinning to a single core. This entire thread is interesting because it is highlighting a similar problem with my virtualized router. When I pinned the router vm to specific cpus the problem goes away. I switched to openstack which doesn’t give me the best control over cpu capabilities and the problem has manifested in a worse form. My uninformed opinion is that there are…

Ah ! Thanks for this. Being single threaded does not prevent you from having your process being migrated from one core to another though, no? Or do you mean that pcap captures are pinned?