Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

bitmagnet.io

Post reply on HN
Print view View original

31–40 of 111 posts

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#31

Post by fluoridation »

latchkey wrote: post #29
fluoridation wrote: post #8

Earlier quoted context omitted.

The way to filter out garbage is to download things with lots of seeds, and if you still happen to download garbage, to immediately stop sharing it.

Chicken/egg problem... as mentioned by someone else above... https://news.ycombinator.com/item?id=37779341 > New releases of something that just came out tend to suffer from this, though. Sometimes the counterfeits reach escape velocity - the rate of people joining in downloading the counterfeit exceed the rate of people realizing and stopping, thus giving the illusion of a legit torrent.

It's possible. I never follow new releases. But back in the ed2k days, I'd say about half of just about any file you cared you search for was fake, regardless of age.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#32

Post by the8472 »

fallat wrote: post #9

It'd be so nice to have a super simple DHT crawler CLI tool, in both implementation and interface.

These things need uptime of hours and days to do it properly and also to stay up to date. There are millions of nodes and torrents and to be non-abusive you have to issue requests at a somewhat sedate pace. And activity kind of moves with the sun due to people who run torrent clients on their home machines. And there are lots of buggy or malicious implementations out there that you have to deal with. So you'd want to run it as a daemon. The CLI would have to be a frontend to the daemon or its database. The UI could be simple. I'm skeptical whether an implementation could be both good and simple.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#33

Post by colinsane »

EGreg wrote: post #11
Akashic101 wrote: post #4

Earlier quoted context omitted.

Considering the screenshot on the linked page shows The Flash among other movies I dont think the author is too concerned about that. I believe its similar to the area that Plex and Jellyfin operate in, mainly that they just provide the framework and tools, what the user does with them is not in their control

How will governments police it when MaidSAFE and other systems distribute totally encrypted content?

BitTorrent already supports encrypted peer connections.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#34

Post by the8472 »

e12e wrote: post #27
Fnoord wrote: post #18

See also: Magnetissimo [1], Torrentinim [2], and Spotweb [3]. [1] https://github.com/sergiotapia/magnetissimo [2] https://github.com/sergiotapia/torrentinim [3] https://github.com/spotweb/spotweb

Any comments on how these compare? Especially in relation to sibling comment about BEP51?

From a brief look at each it seems like they're scraping things like torrent websites, usenet or maybe RSS feeds. Not the DHT.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#35

Post by diggan »

e12e wrote: post #27
Fnoord wrote: post #18

See also: Magnetissimo [1], Torrentinim [2], and Spotweb [3]. [1] https://github.com/sergiotapia/magnetissimo [2] https://github.com/sergiotapia/torrentinim [3] https://github.com/spotweb/spotweb

Any comments on how these compare? Especially in relation to sibling comment about BEP51?

From https://github.com/bitmagnet-io/bitmagnet/issues/11

> The DHT implementation was largely borrowed from Magnetico (https://github.com/boramalper/magnetico) a popular and widely used app which is likewise unable to service these requests.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#36

Post by derefr »

the8472 wrote: post #32
fallat wrote: post #9

It'd be so nice to have a super simple DHT crawler CLI tool, in both implementation and interface.

These things need uptime of hours and days to do it properly and also to stay up to date. There are millions of nodes and torrents and to be non-abusive you have to issue requests at a somewhat sedate pace. And activity kind of moves with the sun due to people who run torrent clients on their home machines. And there are lots of buggy or malicious implementations out there that you have to deal with. So you'd want to…

That's if you're imagining a single node to discover the whole DHT. What if you want to fire off a map-reduce of limited-run DHT explorations starting from different DHT ring positions, where each agent just crawls and emits what it finds on stdout as it finds it?

(In a sense, I suppose this would still be a "daemon", but that daemon would be the map-reduce infrastructure.)

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#38

Post by chmod775 »

> The DHT crawler is not quite unique to bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature.

Heh. That was one of my first projects when I was still learning to code back in 2012: https://github.com/laino/shiny-adventure

The DHT crawler/worker lived seperately, and I eventually put it here to rescue it from a dying HDD: https://github.com/laino/DHT-Torrent-database-Worker

The code is abhorrent and you absolutely shouldn't use it, but it worked. At least the crawler did - the frontend was never completed.

Since the first implementation of mainline DHT appeared in 2005 and crawling that network is really quite an obvious idea, I doubt we (a friend was working on it as well) were first either.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#39

Post by zolland »

the8472 wrote: post #25

It seems like every single of these things always cut corners and don't implement proper, spec-compliant nodes that provide the same services as they use. You know, the "peer" in p2p. BEP51 was designed to make it easier to not trample on the commons, and yet...

Without the peers all we have is to!
Post reply on HN