Boosting upload speed and improving Windows' TCP stack
1–10 of 127 posts
Re: Boosting upload speed and improving Windows' TCP stack
#2I got excited when I saw that fancy Microsoft Message Analyzer tool and wanted to try it out. Sadly it appears to be retired and removed by MSFT? Sad!
Re: Boosting upload speed and improving Windows' TCP stack
#3Interesting. Is the Dropbox client still an obfuscated python app? I'm curious if they spawn new processes for simultaneous uploads since they probably aren't threading.
Re: Boosting upload speed and improving Windows' TCP stack
#4I had a similar issue with Windows kernels "recently" (2016~?)...
I don't have the memory or patience to write a long and inspiring blog post, but it comes down to:
Even with IOCP/multiple threads: network traffic is single threaded in the kernel, even worse, there's a mutex there. Putting the effective limit on PPS for windows to something like 1.1M for 3.0GHz.
The task of this machine was /basically/ a connection multiplexer with some TLS offloading; so listen on a socket, get an encrypted connection, check your connection pool and forward where appropriate.
Our machine basically sat waiting (in kernel space) for this lock 99.7% of the time, 0.3% was spent on SSL handshaking..
We solved our "issue" by spreading such load over many more machines and gave them low-core-count high-clock-speed Xeons instead of the normal complement of 20vCPU Xeons.
AFAIK that issue persists, I'd be interested to know if someone else managed to coerce windows to do the right thing here.
Re: Boosting upload speed and improving Windows' TCP stack
#5Interesting. Is the Dropbox client still an obfuscated python app? I'm curious if they spawn new processes for simultaneous uploads since they probably aren't threading.
> On one hand, Dropbox Desktop Client has just a few settings. On the other, behind this simple UI lies some pretty sophisticated Rust code with multi-threaded compression, chunking, and hashing. On the lowest layers, it is backed up by HTTP/2 and TLS stacks.
And I found another Dropbox blog post about rewriting their sync engine from Python to Rust: https://dropbox.tech/infrastructure/rewriting-the-heart-of-o...
But it isn't clear whether the outer shell of the app might still be Python.
Re: Boosting upload speed and improving Windows' TCP stack
#6I had a similar issue with Windows kernels "recently" (2016~?)... I don't have the memory or patience to write a long and inspiring blog post, but it comes down to: Even with IOCP/multiple threads: network traffic is single threaded in the kernel, even worse, there's a mutex there. Putting the effective limit on PPS for windows to something like 1.1M for 3.0GHz. The task of this machine was /basically/ a connection m…
Do you know whether it's a single thread for all network devices, or just per device? It would be interesting if this ended up being a driver level constraint or something that can be fixed by having multiple NICs in the machine.
Re: Boosting upload speed and improving Windows' TCP stack
#7Interesting. Is the Dropbox client still an obfuscated python app? I'm curious if they spawn new processes for simultaneous uploads since they probably aren't threading.
The Windows client I have installed appears to be a native app using QT 5 and QT5WebEngine (embedded chromium) with an absolutely bonkers number of threads (240). It's possible there's still python in there but I suspect not, their UI has been completely overhauled since the python days.
Re: Boosting upload speed and improving Windows' TCP stack
#8Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all.
I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync.
DB: 500 KiB/s
GD: 3 MiB/s
OD: 11 MiB/s (my max bandwidth with 100Mbps)
Another funny thing is if you just use the webpage, both GD and DB can reach 100Mbps easily.
Edit: should mention Google's DriveFS can reach max speed too, but it's not available for my personal account (which uses the "Backup and sync for Google" app).
Re: Boosting upload speed and improving Windows' TCP stack
#9Dropbox always publishes such good technical blog posts. And as a user, it's reassuring to see how much they still care about technical excellence.
Re: Boosting upload speed and improving Windows' TCP stack
#10Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all. I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync. DB: 500 KiB/s GD: 3 MiB/s OD: 11 MiB/s (my max bandwidth with 100Mbps) I don't know what causes the disparity here, but I have been annoyed by this for years, and it's the sam…
Google are migrating Backup and Sync to DriveFS soon [0], but you can upgrade right now. Now, I don't remember how I did it, but I do have Drive FS on my personal account.
[0]: https://support.google.com/googleone/answer/10309431#zippy=