Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The BitTorrent Protocol Specification v2

bittorrent.org

Post reply on HN
Print view View original

21–30 of 114 posts

Re: The BitTorrent Protocol Specification v2

#21

Post by silotis »

RJIb8RBYxzAMX9u wrote: post #19
jzelinskie wrote: post #6

This update to the spec is a modest change that's largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-byte length of a SHA1 checksum. They've decided to move forward with SHA256 truncated to 20 bytes to avoid incompatibilities with existing infrastructure such as the Mainline DHT. Beyond the hashing algorithm, some important additions that were previously propos…

> They've decided to move forward with SHA256 truncated to 20 bytes [...] I wonder why not SHA512? It's actually faster to compute than SHA512 on 64-bit architectures.

SHA256 was chosen over SHA512 because the two most popular 64-bit ISAs (x86-64 and ARMv8) both define instruction extensions to provide hardware acceleration of SHA256. Hardware support makes SHA256 much faster than SHA512 in software, even on 64-bit processors.

Re: The BitTorrent Protocol Specification v2

#22

Post by supergreg »

kpcyrd wrote: post #5
lowglow wrote: post #2

Can someone diff the spec from the previous version? What's the changelog? :)

It would be interesting to see how the new version compares to ipfs.

The tree structure seems very similar. It would be nice if torrent clients could interact with ipfs or gain ipfs capabilities. Think torrents that update themselves when the files change (thanks to ipns).

Re: The BitTorrent Protocol Specification v2

#23

Post by RJIb8RBYxzAMX9u »

silotis wrote: post #21
RJIb8RBYxzAMX9u wrote: post #19

Earlier quoted context omitted.

> They've decided to move forward with SHA256 truncated to 20 bytes [...] I wonder why not SHA512? It's actually faster to compute than SHA512 on 64-bit architectures.

SHA256 was chosen over SHA512 because the two most popular 64-bit ISAs (x86-64 and ARMv8) both define instruction extensions to provide hardware acceleration of SHA256. Hardware support makes SHA256 much faster than SHA512 in software, even on 64-bit processors.

Huh, I didn't know that's the case for x86. My quick search suggests that support is still quite limited though, and none of my HW (1~4 yrs old) support it. I suppose by the time the spec is finalized and widely adopted HW-accelerated SHA256 would be, too (and I should switch my zfs checksum back from SHA512/256 to SHA256).

Re: The BitTorrent Protocol Specification v2

#24

Post by phire »

Scaevolus wrote: post #16

1) Chunks don't span files . Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwar…

Per-file metadata increases significantly, but it gets rid of the per piece data (which in bittorrent v1 is 20 bytes of sha1 hash per piece and made up the bulk of the .torrent file).

The .torrent file only stores the merkle tree's root hash for each file, and the torrent client will query it's peers to get the rest of the merkle tree (verifiable against the root hash). The leafs of the merkle tree are the hash of each 16kb block.

Interesting consequences of this:

Piece size isn't baked into the file anymore (and I've seen torrents with 16mb blocks), the client can dynamically chose it's verification piece size by requesting only so many layers of the merkle tree. Or it could skip requesting the tree and verify the whole file at once.

Merkle tree roots will be globally unique. You can scan torrent files for duplicated files and download common files from multiple swarms.

Re: The BitTorrent Protocol Specification v2

#25

Post by computerphage »

phire wrote: post #24
Scaevolus wrote: post #16

1) Chunks don't span files . Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwar…

Per-file metadata increases significantly, but it gets rid of the per piece data (which in bittorrent v1 is 20 bytes of sha1 hash per piece and made up the bulk of the .torrent file). The .torrent file only stores the merkle tree's root hash for each file, and the torrent client will query it's peers to get the rest of the merkle tree (verifiable against the root hash). The leafs of the merkle tree are the hash of ea…

> Or it could skip requesting the tree and verify the whole file at once.

To clarify, this works by the client deterministically reconstructing the tree once they have the whole file, then checking the root's hash, correct?

Re: The BitTorrent Protocol Specification v2

#26

Post by yodsanklai »

Scaevolus wrote: post #16

1) Chunks don't span files . Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwar…

> Chunks don't span files. Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others.

But can't you already download one file? I suppose if a chunk spans two files, you may get a few extra KB of another file you don't want, but it's not noticeable from a user perspective.

Re: The BitTorrent Protocol Specification v2

#27

Post by jacksonsabey »

Scaevolus wrote: post #16

1) Chunks don't span files . Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwar…

would 2 different torrents that contain a file with the same exact physical hash share the same torrent file hash?

if torrent A and B both contain the exact same file, but torrent A only has the first half available, and torrent B has the second half available, could I combine both torrents to download that file? this could help fix old dead torrents or at least make the file searchable elsewhere by it's sha256 for example

Re: The BitTorrent Protocol Specification v2

#28

Post by AbacusAvenger »

RJIb8RBYxzAMX9u wrote: post #23
silotis wrote: post #21

Earlier quoted context omitted.

SHA256 was chosen over SHA512 because the two most popular 64-bit ISAs (x86-64 and ARMv8) both define instruction extensions to provide hardware acceleration of SHA256. Hardware support makes SHA256 much faster than SHA512 in software, even on 64-bit processors.

Huh, I didn't know that's the case for x86. My quick search suggests that support is still quite limited though, and none of my HW (1~4 yrs old) support it. I suppose by the time the spec is finalized and widely adopted HW-accelerated SHA256 would be, too (and I should switch my zfs checksum back from SHA512/256 to SHA256).

I haven't found any Intel hardware with the SHA extensions actually implemented. I was expecting it in Skylake but apparently that's not the case... Not sure when we can expect to see them.

Re: The BitTorrent Protocol Specification v2

#29

Post by phire »

computerphage wrote: post #25
phire wrote: post #24

Earlier quoted context omitted.

Per-file metadata increases significantly, but it gets rid of the per piece data (which in bittorrent v1 is 20 bytes of sha1 hash per piece and made up the bulk of the .torrent file). The .torrent file only stores the merkle tree's root hash for each file, and the torrent client will query it's peers to get the rest of the merkle tree (verifiable against the root hash). The leafs of the merkle tree are the hash of ea…

> Or it could skip requesting the tree and verify the whole file at once. To clarify, this works by the client deterministically reconstructing the tree once they have the whole file, then checking the root's hash, correct?

Yeah, it's a deterministic tree of hashes.

Each leaf is the hash of a 16KB chunk. On the next layer up you have a series of nodes which are the hashes of the two leafs below it hashed together.

You add enough layers until you get a single root hash at the root of the tree.

Re: The BitTorrent Protocol Specification v2

#30

Post by daurnimator »

yodsanklai wrote: post #26
Scaevolus wrote: post #16

1) Chunks don't span files . Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwar…

> Chunks don't span files. Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. But can't you already download one file? I suppose if a chunk spans two files, you may get a few extra KB of another file you don't want, but it's not noticeable from a user perspective.

more than a few KB: can be a few MB. It's especially significant if you have a torrent full of text files, and want a subset.
Post reply on HN