Can files be more specifically identified - more than just filename+size? I.e. hash(es)/checksum(s) or something like that?
Show HN: Torrenthistory – an open and legal torrent indexer
11–20 of 31 posts
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#12Can files be more specifically identified - more than just filename+size? I.e. hash(es)/checksum(s) or something like that?
Torrent seed files only contain hashes for "pieces" (typically 1-4MB each), which can cross file boundaries. There's no way to pull an md5 for a specific file without actually snatching it.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#13Why does hosting a magnet link make a torrent index 'illegal'? After all, you're not hosting any actual content on your server.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#14Can files be more specifically identified - more than just filename+size? I.e. hash(es)/checksum(s) or something like that?
Torrent seed files only contain hashes for "pieces" (typically 1-4MB each), which can cross file boundaries. There's no way to pull an md5 for a specific file without actually snatching it.
Pull the files, derive the hashes, and then throw away the files? You only need to retrieve each file once to derive and store the per file hashes.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#15There have been quite a lot of torrent indexers already, but most of them are sooner or later closed. So, I decided to create one more, simple enough (so it can work on its own on a relatively cheap VPS) and legal (so it would be much more difficult to close it). Most of the site data can be freely downloaded. Is there something you think any decent torrent indexer should have?
Could the data be retained on something like IPFS so that cloning of the tracker could happen independent of its existence?
Pin it now!
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#16Can files be more specifically identified - more than just filename+size? I.e. hash(es)/checksum(s) or something like that?
Torrent seed files only contain hashes for "pieces" (typically 1-4MB each), which can cross file boundaries. There's no way to pull an md5 for a specific file without actually snatching it.
BitTorrent v2 also includes a per file hash tree
https://blog.libtorrent.org/2020/09/bittorrent-v2/
so I assume that if you end up with the same file in multiple torrents, you'll be able to grab the files from seeds of these other torrents.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#17Why does hosting a magnet link make a torrent index 'illegal'? After all, you're not hosting any actual content on your server.
That's a very good question. I think that dialog in filesharing/linking court cases has moved from "link vs torrent vs file" issue to "does this site/person make possible bypassing copyright protection, and, if so, to what extent?" issue.
Part of the torrent community still holds to the old (and comfortable, I must say) idea, that files and links (no matter in which form, http or magnet) are something fundamentally different. Files can be illegal, links cannot. Links are just text, infohashes are just numbers and so on.
The reality, as I see it, is different:
1. If something makes bypassing copyright protection as easy as making 2 mouse clicks and waiting a few minutes - that "something" is illegal, the person which operates it also commits a crime. Indirectly, by facilitation (=providing aid), but still.
2. Even if the law in this area is too old or too general to specifically mention such cases - the court practice is still little by little moving in that direction (=files and links doesn't matter, those are technical details; bypassing copyright - does matter)
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#18Earlier quoted context omitted.
Torrent seed files only contain hashes for "pieces" (typically 1-4MB each), which can cross file boundaries. There's no way to pull an md5 for a specific file without actually snatching it.
Pull the files, derive the hashes, and then throw away the files? You only need to retrieve each file once to derive and store the per file hashes.
I though about that, it is technically possible. More than that, it is relatively cheap (incoming traffic on most VPS is free). Still, I decided that searching by file size is much more cost effective. It is almost as good as searching by hash for larger files. It requires little to no code and time to implement. It is available even for torrents without seeds. It doesn't involve downloding files (which, in some people opinion, is illegal as such, even if you don't actually see the downloaded files).
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#19Earlier quoted context omitted.
Torrent seed files only contain hashes for "pieces" (typically 1-4MB each), which can cross file boundaries. There's no way to pull an md5 for a specific file without actually snatching it.
BitTorrent v2 also includes a per file hash tree https://blog.libtorrent.org/2020/09/bittorrent-v2/ so I assume that if you end up with the same file in multiple torrents, you'll be able to grab the files from seeds of these other torrents.
It would be interesting to see if that standard replaces Bittorent v1. For now it is not nearly as popular (everyone just uses v1, I don't know about any site, which has any significant number of v2 torrents).
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#20There have been quite a lot of torrent indexers already, but most of them are sooner or later closed. So, I decided to create one more, simple enough (so it can work on its own on a relatively cheap VPS) and legal (so it would be much more difficult to close it). Most of the site data can be freely downloaded. Is there something you think any decent torrent indexer should have?
Could the data be retained on something like IPFS so that cloning of the tracker could happen independent of its existence?
I thought about some kind of distributed storage, but the technology is not there yet (=I cannot implement that easily, users cannot use it easily). The site uses a regular PostgreSQL database. Most data, though, is published on Internet archive in a standard format (https://torrenthistory.org/docs/open-data/). There are torrents as well, users can download and seed them. I will upload new archives monthly