Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Protocol for Resumable File Uploads

tus.io

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Open Protocol for Resumable File Uploads

#31

Post by irishsultan »

zozbot123 wrote: post #28
treve wrote: post #14

Earlier quoted context omitted.

Responding with 400 Bad Request is actually something that was added after some servers allowed Content-Range on PUT and others didn't. It was never standard, but the end-result was that some clients assumed PUT + Content-Range would work, which meant that some servers would apply the change while others would ignore the header and overwrite the entire resource with the chunk. There's no sane way to add support for t…

> There's no sane way to add support for this header and make older servers behave correctly, so now we have better facilities for this KISS. Endow the "400 Bad Request" server response with a special header that acts like a cookie or nonce, with the semantics "this server does support Content-Range uploads and won't corrupt your resource". If the client resends the PUT + Content-Range request with the correct cookie…

You still need to update all those old servers to stop ignoring the Content-Range and abort in it's presence.

Re: Open Protocol for Resumable File Uploads

#32

Post by kvz »

eps wrote: post #30
kvz wrote: post #21

Earlier quoted context omitted.

Yes, APPEND is not an official HTTP method though. Allowing to modify parts at any location makes things a little bit more complex and comes with some overhead. If you do need to upload multiple chunks simultaneously, you can opt into our Concat extension however, which does exactly that. Our latest blog posts has some images to illustrate.

What overhead is that exactly? My point is that you appear to be pushing for adoption of an extension that handles one specific use case for PATCH, when a more general extension is trivially possible with little to no extra effort.

(I hope I understand your proposal correctly, I fear I might not, if so please clarify, but) more chunks come at the expense of more requests. After a connection drop each separate chunk needs to be renegotiated and transmitted. For some use cases that trade off is well worth it, like when latency is low, but tcp settings or QoS policies won’t let you saturate single connections, so tus does offer ‘sending multiple chunks by default and in parallel’, as an opt-in, via the Concat extension.

If your question is why not make Concat the default mode of operation, the additional roundtrips are the reason. For fragile connections these are often very costly, and we want tus to really shine in those situations, by default. If your users are all operating on big tubes, you’ll likely want to deploy Concat, but that’s not an assumption we want to make.

Re: Open Protocol for Resumable File Uploads

#34

Post by silvestrov »

Tus-Version: 1.0.0,0.2.2,0.2.1

seems like over design. The list will get very long over time.

Just use a single integer instead and have the header include min and max version supported. E.g.

Tus-Version: 1-4

meaning it supports version 1 thru 4. No reason to be able to say version 1 and 4 but not 2 and 3.

Re: Open Protocol for Resumable File Uploads

#35

Post by chrisrhoden »

aiCeivi9 wrote: post #33

Slight Offtopic - why after so many years Chrome & Firefox have so poor support for resuming interrupted file downloads? In case of Firefox I am almost sure it was better in past. I have to use 'wget -c' or https://www.freedownloadmanager.org/ for bigger filles.

As I suspect you may already know, this is dependent on the server 1) indicating support for byte range requests and 2) correctly implementing it.

I don't think I have noticed Firefox getting worse at this over time, but I'm not downloading large files every day. Would you be willing to share where you're noticing this?

Re: Open Protocol for Resumable File Uploads

#36

Post by kvz »

michaelmior wrote: post #29
kvz wrote: post #20

Earlier quoted context omitted.

Yes that is very helpful. Our s3 storage backend for tusd uses it, and our https://uppy.io file uploader does too, usable directly from the browser (so you can choose to not use tus at all with it). S3 resumable uploads do come with a few limitations that make some people still choose tus tho: * chunks need to be >5MB which can be problematic on flaky/poor connections (rural areas, tunnels, clubs/basements, people on…

As far as vendor lock in, it seems like there are a large number of other vendors supporting the S3 API, so this doesn't seem like a huge concern.

That’s a fair point. And I guess with e.g. Minio you could selfhost too.

S3 is great and in fact, at Transloadit we deploy a content ingestion network (reverse cdn) of many regional tusd servers, close to our customers’ end users, but they all ultimately save to S3 using multipart. We’re happy S3 customers.

So why the extra layer. Because this let’s us offer resumability below 5MB, lower regional latencies, roll our own auth, and switch to a different cloud provider without introducing breaking changes at the customer facing side (assuming the new cloud bucket provider does not offer an S3 compatible interface, or even just a slightly incompatible one)

Ultimately you’re still locked-in with AWS protocol-wise, and there’s no community platform for advancing it, so addressing any of these issues is going to be hard.

Re: Open Protocol for Resumable File Uploads

#37

Post by kvz »

silvestrov wrote: post #34

Tus-Version: 1.0.0,0.2.2,0.2.1 seems like over design. The list will get very long over time. Just use a single integer instead and have the header include min and max version supported. E.g. Tus-Version: 1-4 meaning it supports version 1 thru 4. No reason to be able to say version 1 and 4 but not 2 and 3.

We are discussing this very topic here https://github.com/tus/tus-resumable-upload-protocol/issues/... — it has stalled a bit so I would be very happy to see you or other interested/concerned HN readers weigh in. People sharing concerns on GitHub is the main way the protocol has progressed.

Re: Open Protocol for Resumable File Uploads

#38

Post by nikeee »

aiCeivi9 wrote: post #33

Slight Offtopic - why after so many years Chrome & Firefox have so poor support for resuming interrupted file downloads? In case of Firefox I am almost sure it was better in past. I have to use 'wget -c' or https://www.freedownloadmanager.org/ for bigger filles.

It depends on the server, which has to implement HTTP Ranges [0]. Servers like nginx and Apache 2 should suport it. I'm not certain about the whole Node.js and Go backends out there. I think the support in Firefox does not have changed.

[0]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requ...

Re: Open Protocol for Resumable File Uploads

#40

Post by geekodour »

kvz wrote: post #10

Was just casually (well ok maybe it’s more compulsive than that) browsing HN and was pleasantly surprised to find tus on the front page. I’m one of the core contributors and happy to answer questions. Although it’s late here so it may take a few hours while I’m asleep :)

I have not looked into tus properly yet. but how does this compare with bittorrent seeding and can both be combined somehow?
Post reply on HN