Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

All other nodes are operated by someone else, so they don't cooperate on anything beyond what the protocol specifies. Which means everyone is their own little silo. If you want a list of all currently active torrents (millions) then you have to do it with 1 or a handful of nodes, depending on how many IPs you have. DHTs are not arbitrary-distributed-compute frameworks, they're a quite restrictive get/put service.

BEP51[0] does let you query other nodes for a sample of their keys (infohashes) but what they can offer is limited by their vantage point of the network so you need to go around and ask all those millions of nodes. And since it's all random you can't really "search" for anything, you can only sample. And that just gives you 20-byte keys. Afterwards you need to do a lot of additional work to turn those into human-readable metadata.

[0] http://bittorrent.org/beps/bep_0051.html

I ask, because a similar heuristic is used in crypto wallet software, visibility-weighting the various "versions" of a crypto token with the same metadata, by (in part) which were oldest-created. (The logic being: scam clones of a thing need to first observe the real thing, before they can clone it. So the real thing will always come first.)

Of course, I'm assuming here that you're searching for an "expected to exist" release of a thing by a specific distributor, where the distributor has a known-to-you structured naming scheme to the files in their releases, and so you'll only be trying to rank "versions" of the torrent that all have identical names under this naming scheme, save for e.g. the [hash] part of the file name being different to match the content. This won't help if you're trying to find e.g. "X song by Y artist, by any distributor."

> DHT indexing already is possible and done in practice by passively observing get_peers queries. But that approach is inefficient, favoring indexers with lots of unique IP addresses at their disposal. It also incentivizes bad behavior such as spoofing node IDs and attempting to pollute other nodes' routing tables.

If you have a lot of IP addresses (from e.g. AWS Lambda) then you can partition DHT keyspace across a large-N number of nodes and then very quickly discover everything in the keyspace.

The trick is that, since BEP51 exists, you don't need to have all these nodes register themselves into the hash-ring (at arbitrary spoofed positions) to listen. You can just have all these nodes independently probing the hash-ring "from the outside" — just making short-lived connections to registered nodes (without first registering themselves); handshaking that connection as a spoofed node ID; and then firing off one `sample_infohashes` request, getting a response, and disconnectting. The lack of registration shouldn't make any difference, as long as they don't want anyone to try connecting to them.

Which is why I say that these are just "crawler agents", not "nodes" per se. They don't start up P2P at all — to them, this is a one-shot client/server RPC conversation, like a regular web crawler making HTTP requests!

> In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent

Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.