Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Boosting upload speed and improving Windows' TCP stack

dropbox.tech

Post reply on HN
Print view View original

31–40 of 127 posts

Re: Boosting upload speed and improving Windows' TCP stack

#31

Post by whatever_dude »

brundolf wrote: post #9

Dropbox always publishes such good technical blog posts. And as a user, it's reassuring to see how much they still care about technical excellence.

Do they? I constantly see DropBox taking days to sync files that are 30kb on size. Or doing dumbfounding things like download all files, then re-upload all files when I set sync to "online only" on a folder if just one of the files is not set to online only.

Maybe they have grand academic visions and papers, but I've been using them for well over a decade and I feel the client quality has gone downhill over the past few years. They keep adding unnecessary stuff like a redundant file browser while the core service suffers.

Re: Boosting upload speed and improving Windows' TCP stack

#32

Post by jabroni_salad »

chokeartist wrote: post #2

I got excited when I saw that fancy Microsoft Message Analyzer tool and wanted to try it out. Sadly it appears to be retired and removed by MSFT? Sad!

It's really too bad. I'm happy enough to use Wireshark, but I liked that MMA could filter by PID.

Re: Boosting upload speed and improving Windows' TCP stack

#34

Post by betaporter »

dijit wrote: post #4

I had a similar issue with Windows kernels "recently" (2016~?)... I don't have the memory or patience to write a long and inspiring blog post, but it comes down to: Even with IOCP/multiple threads: network traffic is single threaded in the kernel, even worse, there's a mutex there. Putting the effective limit on PPS for windows to something like 1.1M for 3.0GHz. The task of this machine was /basically/ a connection m…

Sounds like you didn't have receive side scaling enabled; by default flows are queued to core 0 to prevent reordering. If you enable RSS, your flows will be hashed to core-specific queues.

It's inaccurate to describe traffic processing as single-threaded in the kernel.

Re: Boosting upload speed and improving Windows' TCP stack

#35

Post by lowleveldesign »

If you’re on recent Windows system, you should have pktmon [1] available. I believe it’s the „netsh trace” successor and has much nicer command line. And you no longer need an external tool to convert the trace to .npcap format.

[1] https://docs.microsoft.com/en-us/windows-server/networking/t...

Re: Boosting upload speed and improving Windows' TCP stack

#36

Post by strictfp »

dijit wrote: post #24
jandrese wrote: post #19

Earlier quoted context omitted.

Did it have to be Windows? This is the sort of thing Linux or *BSD boxes are better suited for. I wouldn't even consider a Windows machine for the task unless there's some sort of licensed software you need to run on it to get the job done.

> This is the sort of thing Linux or *BSD boxes are better suited for. Definitely, though enabling conntrack on Linux has similar characteristics (forces single thread with some kind of internal mutex) though it can do 5x the b/w.. We tried having stateful firewalls in front of our windows boxen, that's how I know. Seems like Cloudflare has an older blog post detailing this too: https://blog.cloudflare.com/conntrack-…

Maybe try wine? Seriously, it might be very low effort to get the binary to run on wine.

Re: Boosting upload speed and improving Windows' TCP stack

#37

Post by strictfp »

encryptluks2 wrote: post #11
thrdbndndn wrote: post #8

Cool article, but I'm not impressed by DropBox's upload speed on my Windows computer, at all. I just tested rn with DropBox, GoogleDrive, and OneDrive, all with their native desktop apps. I simply put a 300MB file in the folder and let it sync. DB: 500 KiB/s GD: 3 MiB/s OD: 11 MiB/s (my max bandwidth with 100Mbps) I don't know what causes the disparity here, but I have been annoyed by this for years, and it's the sam…

Google Drive is a gem. I hope it lasts forever cause no one is competing with them.

Really? Google drive sync has been hot garbage for me. Before that program came everything was fine and dandy, but drive sync constantly stumbles over it's own feet, restarts and fails to up- and download files. I'm longing back to Rsync or even FTP after trying to use google drive to move data.

Re: Boosting upload speed and improving Windows' TCP stack

#38

Post by Isthatablackgsd »

Dylan16807 wrote: post #28
vladvasiliu wrote: post #21

Earlier quoted context omitted.

Is 20 files some kind of hyperbole? If not, how do you get it to do that? I've never seen it transfer more than five files at a time, which sometimes drives me crazy when there are a lot of small files to sync.

It's not hyperbole. I make a backup of my computer, with the output being a bunch of 500MB files. And I would then copy or move those files into a folder on the file stream drive. It's not entirely consistent, and it used to do less, but with some update it decided that it should upload way too many files at once. I've had to switch to an entirely different program to upload those files sequentially.

What is the program that you are using? I am currently using odrive for macOS since they don't have DriveFS support for Apple Silicon. odrive work ok, it just have a weird file conflict sometime.

Re: Boosting upload speed and improving Windows' TCP stack

#39

Post by Dylan16807 »

Isthatablackgsd wrote: post #38
Dylan16807 wrote: post #28

Earlier quoted context omitted.

It's not hyperbole. I make a backup of my computer, with the output being a bunch of 500MB files. And I would then copy or move those files into a folder on the file stream drive. It's not entirely consistent, and it used to do less, but with some update it decided that it should upload way too many files at once. I've had to switch to an entirely different program to upload those files sequentially.

What is the program that you are using? I am currently using odrive for macOS since they don't have DriveFS support for Apple Silicon. odrive work ok, it just have a weird file conflict sometime.

I'm using rclone to do big uploads.

I still use DriveFS for everything else, at least for now. Rclone is capable of mounting the drive but it's not really designed for that.

Re: Boosting upload speed and improving Windows' TCP stack

#40

Post by bob1029 »

stephc_int13 wrote: post #16

Is TCP the best choice? Why not UDP?

This is a good question in my opinion.

Theoretically, UDP would be the best choice if you had the time & money to spend on building a very application-specific layer on top that replicates many of the semantics of TCP. I am not aware of any apps that require 100% of the TCP feature set, so there is always an opportunity to optimize.

You would essentially be saying "I know TCP is great, but we have this one thing we really prefer to do our way so we can justify the cost of developing an in-house mostly-TCP clone and can deal with the caveats of UDP".

If you know your communications channel is very reliable, UDP can be better than TCP.

Now, I am absolutely not advocating that anyone go out and do this. If you are trying to bring a product like Dropbox to market (and you don't have their budget), the last thing you want to do is play games with low-level network abstractions across thousands of potential client device types. TCP is an excellent fit for this use case.

Post reply on HN