Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

bitmagnet.io

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#51

Post by r3trohack3r »

Have been playing around with DHT crawling for a while now, curious how you're getting around the "tiers" of the DHT?

IIUC peers favor nodes they've had longer relationships with to provide stable routes through the DHT.

This means short-lived nodes receive very little traffic, nobody is routing much traffic through fresh nodes, they choose nodes they’ve had longer relationships with.

The longer you stay up, the more you start seeing.

At least this is what I've observed in my projects. The only way I've been able to get anything interesting out of the DHT in the last ~5 years has been to put up a node and leave it up for a long time. If I spin up something, the first day I usually only find a handful of resolvable hashes.

Not to mention it seems the BitTorrent DHT is very lax in what it will route compared to other DHTs (like IPFS) meaning many of the hashes you receive aren't for torrents at all.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#52

Post by JP44 »

chmod775 wrote: post #38

> The DHT crawler is not quite unique to bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. Heh. That was one of my first projects when I was still learning to code back in 2012: https://github.com/laino/shiny-adventure The DHT crawler/worker lived seperately, and I eventually put it here to…

Nothing substantial, I chuckled when I saw the commit history on your linked projects. I do not mean to belittle you (or the purpose/goal of the projects), genuinely enjoyed the distraction and 'results' from it:

Today was the first commit after 11 (9 oct 2012) and 5 years (24 nov 2018), respectively, on the projects. I think your repo might be part of some sort of oldest 'active'- or 'not ported to another repo'-repo

For what I've found in ~10 min (google/gpt), excluding git projects existing before spring 2008 (couldn't get a quick consensus on feb vs april of that year), there's not a lot

(I'll edit this part if sources are requested)

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#53

Post by the8472 »

derefr wrote: post #45
the8472 wrote: post #41

Earlier quoted context omitted.

I don't quite understand what you're proposing here. Generally you only control and operate ~1 node per IPv4 address or per IPv6 /64. All other nodes are operated by someone else, so they don't cooperate on anything beyond what the protocol specifies. Which means everyone is their own little silo. If you want a list of all currently active torrents (millions) then you have to do it with 1 or a handful of nodes, depen…

I mean, what I'm describing is the same thing that BEP51 mentions as a motivation: > DHT indexing already is possible and done in practice by passively observing get_peers queries. But that approach is inefficient, favoring indexers with lots of unique IP addresses at their disposal. It also incentivizes bad behavior such as spoofing node IDs and attempting to pollute other nodes' routing tables. If you have a lot of…

Oh, I already have implemented something[0] like that. It doesn't need lambdas or anything "cloud scale" like that. You "just" need a few dozen to a hundred IP addresses assigned to one machine and run a multi-homed DHT node on that to passively observe traffic from multiple points in the keyspace.

But neither of these approaches is what I'd call a "super simple DHT crawler CLI tool" that the initial comment was asking about. BEP51 is intended to make crawling simple enough that it can run on a single home internet connection, but a proper implementation still isn't trivial.

[0] https://github.com/the8472/mldht

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#54

Post by davidcollantes »

gymbeaux wrote: post #49
coolspot wrote: post #42

Please remove copyrighted movies from the screenshot on your website. It provides evidence that this program is designed for violating copyright, which makes DCMA takedown trivial.

I don’t see any screenshots on the website

There was, and were removed.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#55

Post by no_time »

colinsane wrote: post #33
EGreg wrote: post #11

Earlier quoted context omitted.

How will governments police it when MaidSAFE and other systems distribute totally encrypted content?

BitTorrent already supports encrypted peer connections.

Encryption without authentication in this case is as good as XORing all outgoing data with a fixed key. So basically useless...

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#56

Post by rsync »

coolspot wrote: post #42

Please remove copyrighted movies from the screenshot on your website. It provides evidence that this program is designed for violating copyright, which makes DCMA takedown trivial.

Please determine if these images fall under fair-use provisions and, if so, leave them in place.

Bad actors - whoever they may be - need to see your rights constantly reasserted.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#57

Post by Scion9066 »

LeoPanthera wrote: post #47

> Pipe dream features > In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.

It might be able to do this with the Bittorent V2 format as the hashes would be per file.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#58

Post by miffe »

LeoPanthera wrote: post #47

> Pipe dream features > In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.

dc++ does it, but it's not torrent based.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#59

Post by colinsane »

no_time wrote: post #55
colinsane wrote: post #33

Earlier quoted context omitted.

BitTorrent already supports encrypted peer connections.

Encryption without authentication in this case is as good as XORing all outgoing data with a fixed key. So basically useless...

yes, exactly. BitTorrent supports encryption. swapping it out for some other encryption mechanism won't change anything when it comes to government policing because that's already not where the weaknesses lie for those sharing p2p content today. so what was GP's point?

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#60

Post by complianceowl »

Guys, please educate me: I want to use torrents, but the thought of downloading something inappropriate by clicking on a deceptive link terrifies me (e.g., a download with the title of an action movie, but it turns out to be something else).

How do you guys handle that risk?

Post reply on HN