Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitTorrent file system

github.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: BitTorrent file system

#21

Post by rprospero »

mwfj wrote: post #16

Most use cases or apps don't deal well with read latencies in the thousands of seconds... From quickly reading the source code; it doesn't do any particularly clever optimizations. I'd file this in the "IP over pigeons" category.

I agree that the standard torrent use case of torrents containing small numbers of huge files rather fails with this protocol. However, I think it could be interesting for systems with large numbers of small files.

For instance, think about a LaTeX install. The user could, in a matter of seconds, download a torrent file to a multiple gigabyte repository of all of the fonts an modules that are available. They then run latex on whatever document that they're trying to render. The filesystem would download the libraries as needed, so everything the user needs would be available, but she wouldn't have to wait while downloading a bunch of libraries that she didn't need. Nor would she need to decide before hand which libraries to install or ignore, as everything would be on the filesystem. The user experience of waiting for the executables and base libraries to download whne run for the first time wouldn't be great, but arguable better than waiting for the entire distro to install in the standard case.

Re: BitTorrent file system

#22

Post by joepvd »

rprospero wrote: post #21
mwfj wrote: post #16

Most use cases or apps don't deal well with read latencies in the thousands of seconds... From quickly reading the source code; it doesn't do any particularly clever optimizations. I'd file this in the "IP over pigeons" category.

I agree that the standard torrent use case of torrents containing small numbers of huge files rather fails with this protocol. However, I think it could be interesting for systems with large numbers of small files. For instance, think about a LaTeX install. The user could, in a matter of seconds, download a torrent file to a multiple gigabyte repository of all of the fonts an modules that are available. They then run…

Or... Mounting root as btfs. :)

Re: BitTorrent file system

#23

Post by johansch »

rprospero wrote: post #21
mwfj wrote: post #16

Most use cases or apps don't deal well with read latencies in the thousands of seconds... From quickly reading the source code; it doesn't do any particularly clever optimizations. I'd file this in the "IP over pigeons" category.

I agree that the standard torrent use case of torrents containing small numbers of huge files rather fails with this protocol. However, I think it could be interesting for systems with large numbers of small files. For instance, think about a LaTeX install. The user could, in a matter of seconds, download a torrent file to a multiple gigabyte repository of all of the fonts an modules that are available. They then run…

The basic problem is that very few apps are designed with the mindset that file I/O can take some time. So they'll freeze until you get all of the packets. Even if it only takes 2-3 seconds, that's sort of unacceptable.

In this case, a better experience could be had via a web-to-torrent proxy. Browsers are one of the few apps which don't freeze while waiting for remote I/O...

Re: BitTorrent file system

#24

Post by cbhl »

rprospero wrote: post #21
mwfj wrote: post #16

Most use cases or apps don't deal well with read latencies in the thousands of seconds... From quickly reading the source code; it doesn't do any particularly clever optimizations. I'd file this in the "IP over pigeons" category.

I agree that the standard torrent use case of torrents containing small numbers of huge files rather fails with this protocol. However, I think it could be interesting for systems with large numbers of small files. For instance, think about a LaTeX install. The user could, in a matter of seconds, download a torrent file to a multiple gigabyte repository of all of the fonts an modules that are available. They then run…

Having access to a large repository of files without actually downloading the files to your computer already exists at large corporations, but you want a more traditional serving hierarchy in order to get acceptable read/write latencies.

Bittorrent is more useful when 1) local peer transfer speeds exceed the transfer speed from a master copy, and 2) everybody wants an identical copy of the data.

For example, if you want to deploy a new version of Facebook.com to a far away data center, Bittorrent is great (you only have to upload maybe 2-3x the size of the binary to the datacenter, and the thousands of nodes inside the datacenter can share the pieces amongst themselves).

Re: BitTorrent file system

#27

Post by chrissnell »

cbhl wrote: post #24
rprospero wrote: post #21

Earlier quoted context omitted.

I agree that the standard torrent use case of torrents containing small numbers of huge files rather fails with this protocol. However, I think it could be interesting for systems with large numbers of small files. For instance, think about a LaTeX install. The user could, in a matter of seconds, download a torrent file to a multiple gigabyte repository of all of the fonts an modules that are available. They then run…

Having access to a large repository of files without actually downloading the files to your computer already exists at large corporations, but you want a more traditional serving hierarchy in order to get acceptable read/write latencies. Bittorrent is more useful when 1) local peer transfer speeds exceed the transfer speed from a master copy, and 2) everybody wants an identical copy of the data. For example, if you w…

See: https://blueprints.launchpad.net/glance/+spec/glance-bittorr...

Re: BitTorrent file system

#29

Post by jzelinskie »

There are actually a couple of implementations of this. I think that I most recently saw it in Pulsar[0]. I'm kind of glad these have never gotten too popular because to maintain a healthy swarm using the BitTorrent protocol, peers should not be prioritizing chunks sequentially.

[0]: https://github.com/steeve/pulsar

Re: BitTorrent file system

#30

Post by [deleted] »

mwfj wrote: post #16

Most use cases or apps don't deal well with read latencies in the thousands of seconds... From quickly reading the source code; it doesn't do any particularly clever optimizations. I'd file this in the "IP over pigeons" category.

[deleted]
Post reply on HN