Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Protocol for Resumable File Uploads

tus.io

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Open Protocol for Resumable File Uploads

#21

Post by kvz »

eps wrote: post #17

If I read the spec correctly, PATCH method is actually more of APPEND, no? It would seem logical and practical to allow PATCH to modify any part of a resource that is already present on the server and/or to extend it by appending. This would also make the whole thing useful beyond resuming of interrupted uploads, e.g. to allow for rsync-style updating of existing files.

Yes, APPEND is not an official HTTP method though. Allowing to modify parts at any location makes things a little bit more complex and comes with some overhead. If you do need to upload multiple chunks simultaneously, you can opt into our Concat extension however, which does exactly that. Our latest blog posts has some images to illustrate.

Re: Open Protocol for Resumable File Uploads

#22

Post by [deleted] »

zzo38computer wrote: post #13
treve wrote: post #12

Earlier quoted context omitted.

> An origin server that allows PUT on a given target resource MUST send > a 400 (Bad Request) response to a PUT request that contains a > Content-Range header field (Section 4.2 of [RFC7233]), https://tools.ietf.org/html/rfc7231#section-4.3.4

Maybe that should be fixed, then. HTTP PUT with the range specified seem to me it would be sensible.

[deleted]

Re: Open Protocol for Resumable File Uploads

#24

Post by kvz »

ioquatix wrote: post #19

There is a ruby implementation too: https://github.com/janko-m/tus-ruby-server

Love the work that Janko is doing in our ecosystem! There are implementations for most major languages. So a tus server could even just be some php code that you install with composer and add to your existing Apache setup.

Re: Open Protocol for Resumable File Uploads

#25

Post by emersion »

eximius wrote: post #23

Is there a TL;DR? I see the whole spec is there but I don't have time to read it just this second. Does it use anything fancy like fountain codes or does it just renegotiate chunks each time or something else?

You basically just send the offset when you resume the upload.

Re: Open Protocol for Resumable File Uploads

#26

Post by kvz »

eximius wrote: post #23

Is there a TL;DR? I see the whole spec is there but I don't have time to read it just this second. Does it use anything fancy like fountain codes or does it just renegotiate chunks each time or something else?

The latter.

1. The client POSTs, this allocates a unique Location which the server returns and

2. the client saves this (e.g. in localStorage) along with local file identifiers so it can be looked up later and can

3. query that URL to check how many bytes were already received, and then

4. PATCH the remaining bytes

Repeat step 3 & 4 on failures/resumes.

Re: Open Protocol for Resumable File Uploads

#28

Post by zozbot123 »

treve wrote: post #14
zzo38computer wrote: post #13

Earlier quoted context omitted.

Maybe that should be fixed, then. HTTP PUT with the range specified seem to me it would be sensible.

Responding with 400 Bad Request is actually something that was added after some servers allowed Content-Range on PUT and others didn't. It was never standard, but the end-result was that some clients assumed PUT + Content-Range would work, which meant that some servers would apply the change while others would ignore the header and overwrite the entire resource with the chunk. There's no sane way to add support for t…

> There's no sane way to add support for this header and make older servers behave correctly, so now we have better facilities for this

KISS. Endow the "400 Bad Request" server response with a special header that acts like a cookie or nonce, with the semantics "this server does support Content-Range uploads and won't corrupt your resource". If the client resends the PUT + Content-Range request with the correct cookie/nonce added to it, it has acknowledged this semantics in turn, and the upload can now go through. This adds a roundtrip, but it's still trivial compared to what's being proposed here, and keeps the semantics of PATCH open for more complicated cases.

Re: Open Protocol for Resumable File Uploads

#29

Post by michaelmior »

kvz wrote: post #20
chillaxtian wrote: post #18

S3 Multi-Part Upload API can be used to chunk an object into smaller parts, which can succeed or fail independently. https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview....

Yes that is very helpful. Our s3 storage backend for tusd uses it, and our https://uppy.io file uploader does too, usable directly from the browser (so you can choose to not use tus at all with it). S3 resumable uploads do come with a few limitations that make some people still choose tus tho: * chunks need to be >5MB which can be problematic on flaky/poor connections (rural areas, tunnels, clubs/basements, people on…

As far as vendor lock in, it seems like there are a large number of other vendors supporting the S3 API, so this doesn't seem like a huge concern.

Re: Open Protocol for Resumable File Uploads

#30

Post by eps »

kvz wrote: post #21
eps wrote: post #17

If I read the spec correctly, PATCH method is actually more of APPEND, no? It would seem logical and practical to allow PATCH to modify any part of a resource that is already present on the server and/or to extend it by appending. This would also make the whole thing useful beyond resuming of interrupted uploads, e.g. to allow for rsync-style updating of existing files.

Yes, APPEND is not an official HTTP method though. Allowing to modify parts at any location makes things a little bit more complex and comes with some overhead. If you do need to upload multiple chunks simultaneously, you can opt into our Concat extension however, which does exactly that. Our latest blog posts has some images to illustrate.

What overhead is that exactly?

My point is that you appear to be pushing for adoption of an extension that handles one specific use case for PATCH, when a more general extension is trivially possible with little to no extra effort.

Post reply on HN