Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thinking about 'meta' torrent file format

gist.github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Thinking about 'meta' torrent file format

#21

Post by jychang »

predakanga wrote: post #19
TheLoneWolfling wrote: post #17

Earlier quoted context omitted.

Why not? You just have at most two blocks of additional overhead. You would have to have where the file begins and ends within the blocks downloaded, but that's already in the torrent file.

Because the hashes that are stored in the .torrent operate on that unaligned data. In practice, what this means is that you can't verify that two files of the same name and size but at different alignments within the consolidated data stream are identical; you can't compare hashes, can't do anything without first downloading. This opens the door to mass poisoning of swarms without even having to enter them in the fir…

Why do you want to be completely backwards compatible with classic Torrents? Torrent2 can dump some features of classic torrenting, like folder structure, and mandate that each "subtorrenat" is basically a single Torrent1 containing only 1 file and no folder structure.

Re: Thinking about 'meta' torrent file format

#22

Post by jychang »

KMag wrote: post #18
asdfaoeu wrote: post #7

I've actually been thinking about this a bit as well. I think you can just avoid the torrent file completely and use a merkle tree hash like how new torrent files work and then you end up with just one torrent file per file. And have peer acquisition work through DHT Directories would be simple and just a matter of creating a new "file" with hashes and names of the contents like how git directories (extending on this…

I agree that advertising single file Merkle tree roots on the DHT is a good thing, and that one could nicely build git-like directory structures, but why force the leaves of the tree to be singleton torrent files? Why not instead advertise individual files on the DHT by their Merkle tree roots, and put the Merkle tree roots in each entry of the "files" section of the torrent file? This doesn't force re-packaging of e…

> Why not instead advertise individual files on the DHT by their Merkle tree roots, and put the Merkle tree roots in each entry of the "files" section of the torrent file?

You can't do that because torrents aren't file delimited, they are block delimited. You can't check 2 files are the same across torrents without first downloading both torrents.

Re: Thinking about 'meta' torrent file format

#23

Post by enko »

plorkyeran wrote: post #14
zeitg3ist wrote: post #8

Earlier quoted context omitted.

but doesn't this mean that all the other peers would need to manually upgrade their copy of the torrent file?

Yes. It works reasonably well in this specific case because it's such a niche thing (you don't download 1.25 TB of Touhou music if you don't really care about Touhou music), but it doesn't benefit from people who continue to seed things they've long forgotten about.

> 1.25 TB of Touhou music

I am going to have nightmares tonight...

Re: Thinking about 'meta' torrent file format

#24

Post by j_s »

JTon wrote: post #16
j_s wrote: post #15

Earlier quoted context omitted.

I would think this is a failure of the client, which should support compression formats well enough to be able to fish around inside of the compressed file once it got the metadata portion (zip directory or whatever). http://en.wikipedia.org/wiki/Zip_%28file_format%29#Design : A directory is placed at the end of a .ZIP file. This identifies what files are in the .ZIP and identifies where in the .ZIP that file is loca…

This is a great idea! I wonder why it hasn't already been implemented

I think it is because most torrent client developers are focused on the protocol rather than the end-user experience.

Here is a tool that makes it possible to preview video/audio quality by getting the first and last .rar file: http://techzil.com/play-rar-files-without-extracting-uisng-d...

Re: Thinking about 'meta' torrent file format

#25

Post by KMag »

jychang wrote: post #22
KMag wrote: post #18

Earlier quoted context omitted.

I agree that advertising single file Merkle tree roots on the DHT is a good thing, and that one could nicely build git-like directory structures, but why force the leaves of the tree to be singleton torrent files? Why not instead advertise individual files on the DHT by their Merkle tree roots, and put the Merkle tree roots in each entry of the "files" section of the torrent file? This doesn't force re-packaging of e…

> Why not instead advertise individual files on the DHT by their Merkle tree roots, and put the Merkle tree roots in each entry of the "files" section of the torrent file? You can't do that because torrents aren't file delimited, they are block delimited. You can't check 2 files are the same across torrents without first downloading both torrents.

You misunderstand. This has nothing to do with comparing two torrents. Comparing two torrents solves the wrong problem.

Clients that have downloaded all of the data for a single file (but may or may not have downloaded all of the data for the full torrent) have the data for the file and can calculate the Merkle tree root for that file, and advertise availability on the DHT.

Clients with new style torrent files that included Merkle tree roots in file descriptions would then be able to download those files. This has nothing to do with comparing torrent files.

Post reply on HN