Earlier quoted context omitted.
Could the data be retained on something like IPFS so that cloning of the tracker could happen independent of its existence?
Pin it now!
Sorry, I'm new to HN (as author). What do you mean by "pin"?
Show HN: Torrenthistory – an open and legal torrent indexer
21–30 of 31 posts
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#22It would be very nice if you could search based on file name, as you can with btdig.com. If you could somehow provide BRE without melting down, that would definitely put you way ahead of the other DHT index attempts.
At the very beginning (of January) search was done even through filenames, but then:
1. It turned out, that such search requires quite a lot of disk space for index. 2. It is not very useful without showing the actual matched filenames. This requires more code, I didn't have time to create it at the time.
I added it here for tracking, sooner or later that should be implemented: https://github.com/retonato/torrenthistory-public/issues/1
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#23It would be very nice if you could search based on file name, as you can with btdig.com. If you could somehow provide BRE without melting down, that would definitely put you way ahead of the other DHT index attempts.
At the very beginning (of January) search was done even through filenames, but then: 1. It turned out, that such search requires quite a lot of disk space for index. 2. It is not very useful without showing the actual matched filenames. This requires more code, I didn't have time to create it at the time. I added it here for tracking, sooner or later that should be implemented: https://github.com/retonato/torrenthist…
What is BRE? Search by regular expressions?
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#24Earlier quoted context omitted.
At the very beginning (of January) search was done even through filenames, but then: 1. It turned out, that such search requires quite a lot of disk space for index. 2. It is not very useful without showing the actual matched filenames. This requires more code, I didn't have time to create it at the time. I added it here for tracking, sooner or later that should be implemented: https://github.com/retonato/torrenthist…
What is BRE? Search by regular expressions?
Yup, the most resource constrained form (PCRE2 would be the Cadillac-DoS-my-webapp option). I don't know how abnormal my problem is, but I've run into several cases where I needed a file that had a very specific naming structure - but a couple of substrings needed to be masked because that was the information I was looking for. That can easily be handled with two fixed string passes on a local file system, but not so much at the scale we're talking here.
As far as the indexing issues you mentioned, I'm not a python guy, so I can't recommend a drop in solution. But I have indexed pretty massive string datasets, and you definitely want to select an index method that was specifically devised for the key/value datatype you intend to ingest. So hashmaps are probably out :) It would also pay to adapt either the implementation or the data itself. For example, say you had a bunch of font file name you wanted to index (XyzSans.otf, AsdfSans.otf, AsdfSerif.otf, etc): a prefix tree would be a pretty good fit, especially if you reversed all the strings.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#25Earlier quoted context omitted.
What is BRE? Search by regular expressions?
Yup, the most resource constrained form (PCRE2 would be the Cadillac-DoS-my-webapp option). I don't know how abnormal my problem is, but I've run into several cases where I needed a file that had a very specific naming structure - but a couple of substrings needed to be masked because that was the information I was looking for. That can easily be handled with two fixed string passes on a local file system, but not so…
In principle it is possible, but there would be some limitations:
1. Postgresql supports indexing to make regex searches faster from v9.3 (the current version is 13): https://www.postgresql.org/docs/12/pgtrgm.html . However, it makes them faster only in some (simpler) cases, in others we are back to plain old full table scan.
2. Speaking of full table scan, it is not that bad of an option, the average list of file paths for a torrent file is approximately 6 KB, that's 300 GB per 50 mln torrents, that's completely within reach of some VPS providers (like BuyVM). Still, up to a few minutes per one pass.
3. So, unless I find some efficient technology for regex search in large volumes of text, I would be able to implement only "offline" search (submit query, receive link to search results in 5-30 minutes, depending on server load).
4. Another option would be to provide listings of files for downloading as csv files (torrent_id, filepath) and let user to use command line for searching (or some text viewer/editor, though most popular ones still load all contents into ram). Compressed size would be around 1 KB per one torrent file, that's near 50 GB per 50m torrents.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#26Earlier quoted context omitted.
Yup, the most resource constrained form (PCRE2 would be the Cadillac-DoS-my-webapp option). I don't know how abnormal my problem is, but I've run into several cases where I needed a file that had a very specific naming structure - but a couple of substrings needed to be masked because that was the information I was looking for. That can easily be handled with two fixed string passes on a local file system, but not so…
In principle it is possible, but there would be some limitations: 1. Postgresql supports indexing to make regex searches faster from v9.3 (the current version is 13): https://www.postgresql.org/docs/12/pgtrgm.html . However, it makes them faster only in some (simpler) cases, in others we are back to plain old full table scan. 2. Speaking of full table scan, it is not that bad of an option, the average list of file pa…
Yeah, it isn't something that is feasible if you can't tailor the infrastructure to the underlying data. An SQL backend is about as generic as it gets, hence the poor results. Enterprise DB deployments get away with it because they can justify throwing a lot more hardware behind a centralized generic data store.
Give that second link a closer look if you change your mind. It demonstrates a way of indexing 1.6 billion 80B strings in 24GB, and then returning the result of a fixed string search in 100ms. That is the reward you get when you venturing outside of the LAMP stack: less resource consumption, greater performance, increased utility.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#27Earlier quoted context omitted.
In principle it is possible, but there would be some limitations: 1. Postgresql supports indexing to make regex searches faster from v9.3 (the current version is 13): https://www.postgresql.org/docs/12/pgtrgm.html . However, it makes them faster only in some (simpler) cases, in others we are back to plain old full table scan. 2. Speaking of full table scan, it is not that bad of an option, the average list of file pa…
Yeah, it isn't something that is feasible if you can't tailor the infrastructure to the underlying data. An SQL backend is about as generic as it gets, hence the poor results. Enterprise DB deployments get away with it because they can justify throwing a lot more hardware behind a centralized generic data store. Give that second link a closer look if you change your mind. It demonstrates a way of indexing 1.6 billion…
Thank you, I will
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#28Why does hosting a magnet link make a torrent index 'illegal'? After all, you're not hosting any actual content on your server.
That's a very good question. I think that dialog in filesharing/linking court cases has moved from "link vs torrent vs file" issue to "does this site/person make possible bypassing copyright protection, and, if so, to what extent?" issue. Part of the torrent community still holds to the old (and comfortable, I must say) idea, that files and links (no matter in which form, http or magnet) are something fundamentally d…
So even showing the infohash would be illegal? That's all you really need.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#29Earlier quoted context omitted.
That's a very good question. I think that dialog in filesharing/linking court cases has moved from "link vs torrent vs file" issue to "does this site/person make possible bypassing copyright protection, and, if so, to what extent?" issue. Part of the torrent community still holds to the old (and comfortable, I must say) idea, that files and links (no matter in which form, http or magnet) are something fundamentally d…
So even showing the infohash would be illegal? That's all you really need.
In my opinion even showing infohash would be illegal. Anything, which provides enough information for downloading by a generic torrent client.
Re: Show HN: Torrenthistory – an open and legal torrent indexer
#30Thanks again for all feedback. I added all recent feature suggestions on github (https://github.com/retonato/torrenthistory-public/issues). You can vote for them there (just give a thumbs up for the issue you like) and see the current priority here: https://torrenthistory.org/vote/