Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tus.io: Open Protocol for Resumable File Uploads

tus.io

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Tus.io: Open Protocol for Resumable File Uploads

#21

Post by _urga »

It will need to monitor and induce at most 20-50ms bufferbloat when uploading, regardless of whether the traffic is high or low priority.

Try this demo for yourself:

1. Run "ping google.com" from another computer on your LAN.

2. Upload a 10-20MB file via Gmail or Dropbox from your computer.

3. Watch the ping times on the other computer skyrocket from around 100ms to upwards of 5-10 seconds.

4. Try a Google search from any other computer on your LAN while this is happening.

As an example, Apple's software update actually uses a variant of LEDBAT (the delay sensitive congestion avoidance algorithm from BitTorrent's protocol) when downloading software updates to avoid inducing bufferbloat in the downlink.

Re: Tus.io: Open Protocol for Resumable File Uploads

#22

Post by kvz »

thescriptkiddie wrote: post #10

Resumable file transfers in JS is really cool. But why a totally new protocol? Why not use bittorrent, rsync, or even ftp? A lot of effort has gone into making those reliable in the face of network congestion. Trying to shove a lot of bits down a single TCP connection with no traffic shaping is just asking for bufferbloat to strike.

Hi, one of the authors here. Thanks for raising a valid concern. Vimeo in fact already offers FTP uploads for premium users, but helped us write this as the advantages of something HTTP based are significant. We can:

- be sure that browsers and servers already speak it (so small additional libraries required on both ends)

- gracefully degrade to regular form POSTs should tus support not be possible (IE8, etc)

- easily write hooks for progress bars, and e.g. encoding of the uploaded material

- use existing holes in firewalls. FTP requires many ports (for PORT, DATA, PASV) that are blocked on airports, public libraries, large corporations

- add existing HTTP components to make tus better (loadbalancing, intrusion detection, auth, proxies, etc). Some of these apply to FTP as well, but the HTTP options are more (advanced).

Bittorrent is not really client-server oriented, which is what this protocol is trying to solve. Rsync has many the same disadvantages as FTP listed above.

As for a single TCP connection with no traffic shaping, tus support splitting up a file into multiple parts and uploading them in parallel (as regular tus uploads, so profiting from checksums, retries, and resumability) and stitching them together on the server-side by issuing a 'concat'. This means you can have as many connections as you like, and the individual chunks can complete in any order as well.

Traffic shaping is not part of the protocol. By default tus should saturate available connections.

That said we're writing down some recommendations in a separate 'developer guide' document that offers best practices for implementing and deploying tus, traffic shaping could be part of it. Feel free to weigh in here https://github.com/tus/tus-resumable-upload-protocol/pull/68

Re: Tus.io: Open Protocol for Resumable File Uploads

#23

Post by kvz »

nodesocket wrote: post #9

Do you have to use the tus server written in go ( https://github.com/tus/tusd )? Possible to use a node.js backend?

Hi, one of the authors here. We're currently working on an official implementation in Node.js ES6 now here: https://github.com/tus/tus-node-server - so yes :)

Re: Tus.io: Open Protocol for Resumable File Uploads

#24

Post by kvz »

voltagex_ wrote: post #14

As much as I'd like to see this using an existing standard, I'm very glad - my upload bandwidth is <1 megabit, and I get several disconnections a day (ADSL desync)

Hi, one of the authors here. I really hope the sites that you're using a lot will all implement tus :) I'm confident that could avoid some frustrations

Re: Tus.io: Open Protocol for Resumable File Uploads

#25

Post by kvz »

advisedwang wrote: post #5

If anyone is interested in prior art, take a look at Google Cloud Storage's resumable uploads: https://cloud.google.com/storage/docs/json_api/v1/how-tos/up... It's very domain specific and relies on GCP's JSON API so probably isn't suitable to apply broadly.

Hi, one of the authors here. Indeed, resumable uploads are not a thing we invented. Nevertheless, we think it's handy to have a document describing the approach and client and server implementations. Amazon, Dropbox and Google offer some in the way of describing how they do it, but nothing to address the problems that wider audience could encounter, and no platform for collaboration to make it better. tus was built under MIT and on GitHub and will continue to evolve through contributions from the community (although we consider that what we cemented into 1.0.0 ready for adoption)

Re: Tus.io: Open Protocol for Resumable File Uploads

#26

Post by kvz »

amelius wrote: post #3

Does this allow traffic shaping on the client? Or will this "stuff the pipe" once an upload starts?

Hi, one of the authors here. It would, but traffic shaping is not part of the protocol itself and by default tus will saturate available connections.

That said we're writing down some recommendations in a separate 'developer guide' document that offers best practices for implementing and deploying tus, traffic shaping could be part of it. Feel free to weigh in here https://github.com/tus/tus-resumable-upload-protocol/pull/68

Re: Tus.io: Open Protocol for Resumable File Uploads

#27

Post by kvz »

AReallyGoodName wrote: post #12

How long till you guys get a Java server library out? Currently using AWS API for resumable uploads here.

Hi, one of the authors here. You are the first one to request it, I've documented that here https://github.com/tus/tus-resumable-upload-protocol/issues/....

None of the current core members have experience running Java servers in the real world so it's hard to see what would be involved. We're definitely open to discussing it in the issue if you like.

Re: Tus.io: Open Protocol for Resumable File Uploads

#28

Post by kvz »

daurnimator wrote: post #17

The 100-Continue idea mentioned in the 3 year old top comment seems like a much better idea. The issues on the tus issue tracker seem to discard this prematurely.

Hi, one of the authors here. 100 Continue wasn't discarded that quickly and is part of the protocol. More info here: https://github.com/tus/tus-resumable-upload-protocol/issues/...

Re: Tus.io: Open Protocol for Resumable File Uploads

#29

Post by kvz »

seivan wrote: post #7
cwt137 wrote: post #6

Why are they not prefixing their headers with "X-"?

I recall it became unnecessary at some point(?) Might be relevant. https://tools.ietf.org/html/rfc6648

Hi, one of the authors here. Correct, we added that to the protocol FAQ too: https://github.com/tus/tus-resumable-upload-protocol/blob/ma...

The only exception is X-HTTP-Method-Override, as people have been standardizing on this, and the header is specifically meant to let people deploy tus in environments that don't support all of HTTP and are apparently hard to change.

Re: Tus.io: Open Protocol for Resumable File Uploads

#30

Post by amelius »

kvz wrote: post #26
amelius wrote: post #3

Does this allow traffic shaping on the client? Or will this "stuff the pipe" once an upload starts?

Hi, one of the authors here. It would, but traffic shaping is not part of the protocol itself and by default tus will saturate available connections. That said we're writing down some recommendations in a separate 'developer guide' document that offers best practices for implementing and deploying tus, traffic shaping could be part of it. Feel free to weigh in here https://github.com/tus/tus-resumable-upload-protocol…

I think traffic shaping is very desirable when using a websocket for a separate command-channel. The last thing you want is an upload blocking all communications over that command-channel.

Furthermore, browsers can open only a maximum number of connections at the same time (I believe the maximum on some browsers is even just 2). What if the application already uses one connection for a websocket? Will there be only 1 connection left for the upload? And what if the browser needs to download other resources in the background, such as images, fonts, etc.?

Just some concerns, good luck with the project :)

Post reply on HN