Open Protocol for Resumable File Uploads • Hacker News Board

Re: Open Protocol for Resumable File Uploads

#41

Post by kvz » Fri, Nov 23, 2018, 2:57 PM UTC

geekodour wrote: post #40

kvz wrote: post #10

Was just casually (well ok maybe it’s more compulsive than that) browsing HN and was pleasantly surprised to find tus on the front page. I’m one of the core contributors and happy to answer questions. Although it’s late here so it may take a few hours while I’m asleep :)

I have not looked into tus properly yet. but how does this compare with bittorrent seeding and can both be combined somehow?

People ask that more yes, on the surface they have a lot in common. Both can be used to transmit huge files, both can chunk files up and only transmit remaining parts, and pick up and resume at a later point in time, and (in case of tus optionally with the Concat extension) send these chunks simultaneously.

Tus however works as a thin layer on top of HTTP, so it’s easy to drop into existing web sites/load balancers/auth proxies/firewalls. BitTorrent ports are often closed off on airports/hotels/corporate networks. But websites work. And if you can access a website, you will be able to upload files to it with tus.

Another difference is that tus assumes classic client/server roles. The client uploads to the server. Downloading is done via your regular http stack and not facilitated by tus. BitTorrent facilitates both uploading and downloading in single clients. It is more peer-to-peer and decentralized in nature, where tus clients typically upload to a central point (like: many video producers upload to Vimeo. Not very contrived as Vimeo adopted tus).

There are more differences (Discoverability, trackers, pull vs push, pulling from many peers at once) but the comment is getting very long so I hope this already helps a bit :)

Happy to dive deeper into this at request tho :)

Re: Open Protocol for Resumable File Uploads

#42

Post by michaelmior » Fri, Nov 23, 2018, 3:31 PM UTC

kvz wrote: post #36

michaelmior wrote: post #29

Earlier quoted context omitted.

As far as vendor lock in, it seems like there are a large number of other vendors supporting the S3 API, so this doesn't seem like a huge concern.

That’s a fair point. And I guess with e.g. Minio you could selfhost too. S3 is great and in fact, at Transloadit we deploy a content ingestion network (reverse cdn) of many regional tusd servers, close to our customers’ end users, but they all ultimately save to S3 using multipart. We’re happy S3 customers. So why the extra layer. Because this let’s us offer resumability below 5MB, lower regional latencies, roll our…

My comment wasn't intended to a dig at tus. Like the work you all are doing!

Re: Open Protocol for Resumable File Uploads

#43

Post by aaaaaaaaaab » Fri, Nov 23, 2018, 4:47 PM UTC

zozbot123 wrote: post #28

treve wrote: post #14

Earlier quoted context omitted.

Responding with 400 Bad Request is actually something that was added after some servers allowed Content-Range on PUT and others didn't. It was never standard, but the end-result was that some clients assumed PUT + Content-Range would work, which meant that some servers would apply the change while others would ignore the header and overwrite the entire resource with the chunk. There's no sane way to add support for t…

> There's no sane way to add support for this header and make older servers behave correctly, so now we have better facilities for this KISS. Endow the "400 Bad Request" server response with a special header that acts like a cookie or nonce, with the semantics "this server does support Content-Range uploads and won't corrupt your resource". If the client resends the PUT + Content-Range request with the correct cookie…

Or do a HEAD on the resource you want to resume uploading (this is recommended to find out how many bytes have actually went through) and if the response contains a "Accept-Range: true" header then the client can resume the upload.

Re: Open Protocol for Resumable File Uploads

#44

Post by kvz » Fri, Nov 23, 2018, 7:33 PM UTC

michaelmior wrote: post #42

kvz wrote: post #36

Earlier quoted context omitted.

That’s a fair point. And I guess with e.g. Minio you could selfhost too. S3 is great and in fact, at Transloadit we deploy a content ingestion network (reverse cdn) of many regional tusd servers, close to our customers’ end users, but they all ultimately save to S3 using multipart. We’re happy S3 customers. So why the extra layer. Because this let’s us offer resumability below 5MB, lower regional latencies, roll our…

My comment wasn't intended to a dig at tus. Like the work you all are doing!

<3

Re: Open Protocol for Resumable File Uploads

#45

Post by zzo38computer » Sat, Nov 24, 2018, 2:14 AM UTC

treve wrote: post #14

zzo38computer wrote: post #13

Earlier quoted context omitted.

Maybe that should be fixed, then. HTTP PUT with the range specified seem to me it would be sensible.

Responding with 400 Bad Request is actually something that was added after some servers allowed Content-Range on PUT and others didn't. It was never standard, but the end-result was that some clients assumed PUT + Content-Range would work, which meant that some servers would apply the change while others would ignore the header and overwrite the entire resource with the chunk. There's no sane way to add support for t…

You are right about those things, and some of the proposed solutions (that one and others).

It look to me PATCH is actually better; perhaps one of the patch formats can be a partial patch, for example if the Content-Type of the PATCH request is application/partial-content-patch then the first line of the body is the contents of the Content-Range header. In my opinion, this look better to me than the other replies to the message that this message is in reply to (although I admit anything I write may be mistaken; I am not perfect).