Earlier quoted context omitted.
QUIC is UDP, and TCP does not use CCA in userspace.
QUIC absolutely uses congestion control. See section 6 here https://tools.ietf.org/id/draft-ietf-quic-recovery-26.html
[deleted]
Boosting upload speed and improving Windows' TCP stack
71–80 of 127 posts
Re: Boosting upload speed and improving Windows' TCP stack
#72Earlier quoted context omitted.
QUIC is UDP, and TCP does not use CCA in userspace.
QUIC does run in user space, and also uses congestion controllers running inside the QUIC stack, in user space. (I work on a QUIC implementation in Rust.)
QUIC is a protocol... "...CCA in userspace" CCA stands for congestion control algorithm.
Re: Boosting upload speed and improving Windows' TCP stack
#73Re: Boosting upload speed and improving Windows' TCP stack
#74Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all. I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync. DB: 500 KiB/s GD: 3 MiB/s OD: 11 MiB/s (my max bandwidth with 100Mbps) I don't know what causes the disparity here, but I have been annoyed by this for years, and it's the sam…
Interesting, can you try disabling upload limiter in settings? Also what is your RTT to `nsf-1.dropbox.com`?
PS. One known problem that we have right now is that we use a multiplexed HTTP/2 connection, therefore:
1) We rely on the host's TCP congestion. (We have not yet switched to HTTP/3 w/ BBR.)
2) We currently use a single TCP connection: it is more fair to the other traffic on the link but can become bottleneck on large RTTs.
Re: Boosting upload speed and improving Windows' TCP stack
#75The real root cause for all that flow director mess and core balancing is that there's a huge disconnect between how the hardware works and what the socket API offers by default.
The scaling model of the hardware is rather simple: hash over packet headers and assign a queue based on this. And each queue should be pinned to a core by pinning the interrupts, so you got easy flow-level scaling. That's called RSS. It's simple and effective. What it means is: the hardware decides which core handles which flow. I wonder why the article doesn't mention RSS at all?
Now the socket API works in a different way: your application decides which core handles which socket and hence which flow. So you get cache misses if you don't tak into account how the hardware is hashing your flows. That's bad. So you can do some work-arounds by using flow director to explicitly redirect flows to cores that handle things but that's just not really an elegant solution (and the flow director lookup tables are small-ish).
I didn't follow kernel development regarding this recently, but there should be some APIs to get a mapping from a connection tuple to the core it gets hashed to on RX (hash function should be standardized to Toeplitz IIRC, the exact details on which fields and how they are put into the function are somewhat hardware- and driver-specific but usually configurable). So you'd need to take this information into account when scheduling your connections to cores. If you do that you don't get any cache misses and don't need to rely on the limited capabilities of explicit per-flow steering.
Note that this problem will mostly go away once TAPS finally replaces BSD sockets :)
Re: Boosting upload speed and improving Windows' TCP stack
#76How come Linux doesn't have this issue? Why did Microsoft had to fix TCP with the RACK-TLP RFC when both Linux and MacOS implementations did fine already?
Microsoft Devs explain this in their "Algorithmic improvements boost TCP performance on the Internet"[1] article.
TL;DR is that they had RACK (RFC draft) implemented as an MVP but w/o the reordering heuristic.
[1] https://techcommunity.microsoft.com/t5/networking-blog/algor...
Re: Boosting upload speed and improving Windows' TCP stack
#77Is TCP the best choice? Why not UDP?
We will be eventually migrating to UDP (HTTP/3) once it is rolled out on Envoys[0] on Dropbox Edge Network[1].
[0] https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...
[1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastr...
Re: Boosting upload speed and improving Windows' TCP stack
#78Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all. I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync. DB: 500 KiB/s GD: 3 MiB/s OD: 11 MiB/s (my max bandwidth with 100Mbps) I don't know what causes the disparity here, but I have been annoyed by this for years, and it's the sam…
Interesting, can you try disabling upload limiter in settings? Also what is your RTT to `nsf-1.dropbox.com`? PS. One known problem that we have right now is that we use a multiplexed HTTP/2 connection, therefore: 1) We rely on the host's TCP congestion. (We have not yet switched to HTTP/3 w/ BBR.) 2) We currently use a single TCP connection: it is more fair to the other traffic on the link but can become bottleneck o…
Tried to change upload speed to no limit, doesn't make much difference.
Ping result:
Pinging nsf-env-1.dropbox-dns.com [162.125.3.12] with 32 bytes of data:
Reply from 162.125.3.12: bytes=32 time=27ms TTL=55
Reply from 162.125.3.12: bytes=32 time=27ms TTL=55
Reply from 162.125.3.12: bytes=32 time=27ms TTL=55
Reply from 162.125.3.12: bytes=32 time=27ms TTL=55
Ping statistics for 162.125.3.12:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 27ms, Maximum = 27ms, Average = 27ms
Is the OS being Win7 a factor? (Work computer, can't update [yet]).
Download speed is normal (100Mbps).
Re: Boosting upload speed and improving Windows' TCP stack
#79The real root cause for all that flow director mess and core balancing is that there's a huge disconnect between how the hardware works and what the socket API offers by default. The scaling model of the hardware is rather simple: hash over packet headers and assign a queue based on this. And each queue should be pinned to a core by pinning the interrupts, so you got easy flow-level scaling. That's called RSS. It's s…
We didn't mention RSS/RPS in the post mostly because they are stable. (Albeit, relatively ineffective in terms of L2 cache misses.) FlowDirector, OTOH, breaks that stability and causes a lot of migrations, and hence a lot of re-ordering.
Anyways, nice reference for TAPS! Fo those wanting to dig into it a bit more, consider reading an introductory paper (before a myriad of RFC drafts from the "TAPS Working Group"): https://arxiv.org/pdf/2102.11035.pdf
PS. We went through most of our low-level web-server optimization for the Edge Network in an old blogpost: https://dropbox.tech/infrastructure/optimizing-web-servers-f...
Re: Boosting upload speed and improving Windows' TCP stack
#80I appreciate seeing a support and engineering org going this deep to resolve this kind of issue. Normally this is the stuff you waste hours on with a support org only to get told to clear your cookies and cache one more time.
In particular, the collaboration with Microsoft was great.I wonder what it took to make that happen.