Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thinking about 'meta' torrent file format

gist.github.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Thinking about 'meta' torrent file format

#11

Post by predakanga »

dz0ny wrote: post #2

Private trackers will say no. Public trackers may welcome this...

There are many types of private tracker that would love to see this - for instance consider gaming trackers, where you may have a single .torrent for a large collection of ROMs, or DLC for a game. Consider TV trackers, tracking a whole TV season with a single .torrent file, or music trackers with discographies.

More importantly, a key concern on private trackers is swarm size - an extension like this would have the potential to expand the available peers on a given file, if the file exists in other swarms on the same tracker. Not a very common use case, but one to consider nonetheless.

Re: Thinking about 'meta' torrent file format

#13

Post by predakanga »

brokenparser wrote: post #6

Perhaps we could make trackers more intelligent and have them combine peer pools, so they create something like a venn diagram of torrents. In addition to telling you which peers are available, it'll tell you what to request from them. You already have all of the file hashes in the torrent anyway, so any wrongdoing here will get discarded.

Unfortunately it's not as simple as that - when asking each other for data, the individual peers ask for a particular 'piece' of the torrent, where that piece isn't relative to a given file, but the torrent as a whole.

The files are concatenated into one long stream, and the piece number is an index to that, with no guarantees about alignment.

For instance, if you have a torrent (we'll call it 'X') with three files: the 4mb file 'a', the 3mb file 'b' and the 1mb file 'c', and two separate torrents ('Y' and 'Z') describing files 'b' and 'c' seperately, then the pieces would map something like this:

'Y' piece 1 -> 'X' piece 17 'Z' piece 1 -> 'X' piece 29

That's an absolute best case scenario though - in most cases, file sizes aren't quite as perfect as that (each being a multiple of the default piece size, 256kb). If 'b' just happened to be 1373kb, or anything else that wasn't a multiple of 256kb, then any files after it aren't addressable from other torrents.

Re: Thinking about 'meta' torrent file format

#14

Post by plorkyeran »

zeitg3ist wrote: post #8
oakwhiz wrote: post #4

To see one method that is used to work around this sort of thing: The folks over at http://www.tlmc.eu/ have been expanding the same 1.2TB collection of files for a while, just by stopping the old torrent, running a Python script to patch the changes, and then rechecking and starting the new torrent from the old directory.

but doesn't this mean that all the other peers would need to manually upgrade their copy of the torrent file?

Yes. It works reasonably well in this specific case because it's such a niche thing (you don't download 1.25 TB of Touhou music if you don't really care about Touhou music), but it doesn't benefit from people who continue to seed things they've long forgotten about.

Re: Thinking about 'meta' torrent file format

#15

Post by j_s »

kovalkos wrote: post #10

Another problem with torrents is compression of files. Compressing a torrent makes it impossible to select only 1 file from a big collection.

I would think this is a failure of the client, which should support compression formats well enough to be able to fish around inside of the compressed file once it got the metadata portion (zip directory or whatever).

http://en.wikipedia.org/wiki/Zip_%28file_format%29#Design: A directory is placed at the end of a .ZIP file. This identifies what files are in the .ZIP and identifies where in the .ZIP that file is located. This allows .ZIP readers to load the list of files without reading the entire .ZIP archive.

Re: Thinking about 'meta' torrent file format

#16

Post by JTon »

j_s wrote: post #15
kovalkos wrote: post #10

Another problem with torrents is compression of files. Compressing a torrent makes it impossible to select only 1 file from a big collection.

I would think this is a failure of the client, which should support compression formats well enough to be able to fish around inside of the compressed file once it got the metadata portion (zip directory or whatever). http://en.wikipedia.org/wiki/Zip_%28file_format%29#Design : A directory is placed at the end of a .ZIP file. This identifies what files are in the .ZIP and identifies where in the .ZIP that file is loca…

This is a great idea! I wonder why it hasn't already been implemented

Re: Thinking about 'meta' torrent file format

#17

Post by TheLoneWolfling »

predakanga wrote: post #13
brokenparser wrote: post #6

Perhaps we could make trackers more intelligent and have them combine peer pools, so they create something like a venn diagram of torrents. In addition to telling you which peers are available, it'll tell you what to request from them. You already have all of the file hashes in the torrent anyway, so any wrongdoing here will get discarded.

Unfortunately it's not as simple as that - when asking each other for data, the individual peers ask for a particular 'piece' of the torrent, where that piece isn't relative to a given file, but the torrent as a whole. The files are concatenated into one long stream, and the piece number is an index to that, with no guarantees about alignment. For instance, if you have a torrent (we'll call it 'X') with three files:…

Why not?

You just have at most two blocks of additional overhead.

You would have to have where the file begins and ends within the blocks downloaded, but that's already in the torrent file.

Re: Thinking about 'meta' torrent file format

#18

Post by KMag »

asdfaoeu wrote: post #7

I've actually been thinking about this a bit as well. I think you can just avoid the torrent file completely and use a merkle tree hash like how new torrent files work and then you end up with just one torrent file per file. And have peer acquisition work through DHT Directories would be simple and just a matter of creating a new "file" with hashes and names of the contents like how git directories (extending on this…

I agree that advertising single file Merkle tree roots on the DHT is a good thing, and that one could nicely build git-like directory structures, but why force the leaves of the tree to be singleton torrent files?

Why not instead advertise individual files on the DHT by their Merkle tree roots, and put the Merkle tree roots in each entry of the "files" section of the torrent file? This doesn't force re-packaging of existing torrents into singleton torrents. Seeders can advertise single files from old torrents and clients with new torrents can take advantage of this advertising.

I disagree with the munged-key darknet idea. If you want a darknet, run it on a non-public DHT, with cryptographic handshakes and encrypted traffic. Cryptographically munging the DHT keys on a public DHT only creates a "light grey net" that's trivially circumvented and provides a false sense of privacy.

Re: Thinking about 'meta' torrent file format

#19

Post by predakanga »

TheLoneWolfling wrote: post #17
predakanga wrote: post #13

Earlier quoted context omitted.

Unfortunately it's not as simple as that - when asking each other for data, the individual peers ask for a particular 'piece' of the torrent, where that piece isn't relative to a given file, but the torrent as a whole. The files are concatenated into one long stream, and the piece number is an index to that, with no guarantees about alignment. For instance, if you have a torrent (we'll call it 'X') with three files:…

Why not? You just have at most two blocks of additional overhead. You would have to have where the file begins and ends within the blocks downloaded, but that's already in the torrent file.

Because the hashes that are stored in the .torrent operate on that unaligned data.

In practice, what this means is that you can't verify that two files of the same name and size but at different alignments within the consolidated data stream are identical; you can't compare hashes, can't do anything without first downloading. This opens the door to mass poisoning of swarms without even having to enter them in the first place.

There are potential solutions (including providing a broader hash per-file, as opposed to per-piece), but my statement was only that it's not that simple, not that it's impossible.

Re: Thinking about 'meta' torrent file format

#20

Post by [deleted] »

predakanga wrote: post #19
TheLoneWolfling wrote: post #17

Earlier quoted context omitted.

Why not? You just have at most two blocks of additional overhead. You would have to have where the file begins and ends within the blocks downloaded, but that's already in the torrent file.

Because the hashes that are stored in the .torrent operate on that unaligned data. In practice, what this means is that you can't verify that two files of the same name and size but at different alignments within the consolidated data stream are identical; you can't compare hashes, can't do anything without first downloading. This opens the door to mass poisoning of swarms without even having to enter them in the fir…

[deleted]
Post reply on HN