BEP 30: Merkle hash torrent extension
bittorrent.org
BEP 30: Merkle hash torrent extension
1–10 of 28 posts
Re: BEP 30: Merkle hash torrent extension
#2Isn't this how git works? I would have thought bit torrent did this all along.
Re: BEP 30: Merkle hash torrent extension
#3Hash trees are pretty cool. For some fancier uses, see the Peer to Peer Streaming Protocol (PPSP), which is authored by the libswift/tribler guys.
https://datatracker.ietf.org/wg/ppsp/documents/
Basically, instead of the stream source having to sign every single new chunk (so peers can verify that they're getting the right data), the source signs subtree hashes of the new data and slowly builds up a larger hash tree. Once the stream is over, the complete hash tree is instantly seedable by anybody in the original stream.
Re: BEP 30: Merkle hash torrent extension
#4Very similar to IPFS's merkle DAG and seems like a critical element of getting content centric networking projects like Bitorrent, Inc's Project Malestrom up and running
Re: BEP 30: Merkle hash torrent extension
#5ipfs uses exactly the same strategy to distribute content via the hash of their root merkleDAG node. http://ipfs.io
Re: BEP 30: Merkle hash torrent extension
#6Isn't this how git works? I would have thought bit torrent did this all along.
[deleted]
Re: BEP 30: Merkle hash torrent extension
#7Hash trees are pretty cool. For some fancier uses, see the Peer to Peer Streaming Protocol (PPSP), which is authored by the libswift/tribler guys. https://datatracker.ietf.org/wg/ppsp/documents/ Basically, instead of the stream source having to sign every single new chunk (so peers can verify that they're getting the right data), the source signs subtree hashes of the new data and slowly builds up a larger hash tree.…
The (other?) big win of pspp is the ability to stream live feeds.
Re: BEP 30: Merkle hash torrent extension
#8Isn't this how git works? I would have thought bit torrent did this all along.
Not exactly. Git uses per-file, per-commit, and per-directory hashes. It does make up a sort of hash tree, but the tree does not descend within a given file. You need to hash the entire file and determine its hash to know if any of the pieces you have are valid. This would be a problem for large files if many of those pieces come from untrusted sources -- you'd have to spend a lot of time/bandwidth downloading invalid data before you realized it. This isn't generally the case for Git, but it for BitTorrent.
BitTorrent traditionally solves this using a hash list. All of the data in a torrent is broken up between pieces of a chosen size, and hashes are calculated for each of those pieces individually. (These are generally not aligned with file boundaries, which is why you may have noticed that even if you tell your torrent to only download a certain set of files, you may still end up with some data from adjacent files.)
This entirely hash list is included in the torrent file. For torrents with a lot of data, this could be 10MB or more.
If you're using a magnet link, that torrent file needs to be downloaded from peers. This brings back the original problem: you need to download this entire large file before you know that the peer isn't just sending you random data.
BEP-30 proposes a solution: generate a binary hash tree whose leaves are the torrent pieces, and include only the single root hash in the torrent file to keep it small. When you're getting pieces from a peer, they send you the missing inner hashes of the tree that you need to verify the piece.
The minimum data transfer in ideal circumstances is increased a bit, but the peer-to-peer system is made more robust, able to identify invalid data much faster.
I think it's a great modification to the protocol. Unfortunately it isn't widely-enough supported to be practical for general use.
Re: BEP 30: Merkle hash torrent extension
#9"Large torrent files put a strain on the Web servers distributing them".
Finally! bittorrent designers are acknowledging the centralized deficiency in the torrent protocol and implementing Merkle / root hash distribution model, as eDonkey / eMule used since 2000's:
https://en.wikipedia.org/wiki/Ed2k_URI_scheme#eD2k_hash_algo...
15 years later, everything old is new again.
Re: BEP 30: Merkle hash torrent extension
#10"Large torrent files put a strain on the Web servers distributing them". Finally! bittorrent designers are acknowledging the centralized deficiency in the torrent protocol and implementing Merkle / root hash distribution model, as eDonkey / eMule used since 2000's: https://en.wikipedia.org/wiki/Ed2k_URI_scheme#eD2k_hash_algo... 15 years later, everything old is new again.
>Last-Modified: Fri Aug 28 18:30:47 2009 +0000
9 years.