Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

bitmagnet.io

Post reply on HN
Print view View original

11–20 of 111 posts

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#11

Post by EGreg »

Akashic101 wrote: post #4
thelastparadise wrote: post #2

Very interesting. Does this approach actually work in practice? Also what happens if illegal content gets scooped up into the index?

Considering the screenshot on the linked page shows The Flash among other movies I dont think the author is too concerned about that. I believe its similar to the area that Plex and Jellyfin operate in, mainly that they just provide the framework and tools, what the user does with them is not in their control

How will governments police it when MaidSAFE and other systems distribute totally encrypted content?

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#12

Post by no_time »

lobsterslive wrote: post #5
thelastparadise wrote: post #2

Very interesting. Does this approach actually work in practice? Also what happens if illegal content gets scooped up into the index?

It works well in practice. The DHT protocol includes announce messages that broadcast when new files are shared on BitTorrent. It then includes a "geometric" way to find people who are sharing those files. It doesn't include the files themselves, just the torrents which include a file list and location hashes. If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over ti…

>If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over time),

Correct me if I'm wrong but as far as I understand, passively listening on DHT would only mean you build up a list of infohashes of everything shared on BitTorrent. You'd actually have to reach out to your DHT peers to know what files the infohashes actually represents.

Wrapping back to grandparent's question of

>Also what happens if illegal content gets scooped up into the index?

I think this could get dicey if someone announces something very illegal like CP, and your crawler starts asking every peer that announced the infohash about it's contents with this[0] protocol. This would put your IP into a pretty awful exclusive club of

A, other crawlers

B, actual people wanting downloading said CP

[0]: https://www.bittorrent.org/beps/bep_0009.html

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#13

Post by kevincox »

hypertele-Xii wrote: post #10
grepfru_it wrote: post #6

Earlier quoted context omitted.

>It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually h…

While you're technically correct, the protocol is resilient to such attack, as the number of people participating in a particular torrent is a good indicator of its validity. After all, everyone who was fooled will delete and stop sharing such items. New releases of something that just came out tend to suffer from this, though. Sometimes the counterfeits reach escape velocity - the rate of people joining in downloadi…

It seems like it would be pretty easy to make it appear that your spam torrent is highly active.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#14

Post by grepfru_it »

hypertele-Xii wrote: post #10
grepfru_it wrote: post #6

Earlier quoted context omitted.

>It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually h…

While you're technically correct, the protocol is resilient to such attack, as the number of people participating in a particular torrent is a good indicator of its validity. After all, everyone who was fooled will delete and stop sharing such items. New releases of something that just came out tend to suffer from this, though. Sometimes the counterfeits reach escape velocity - the rate of people joining in downloadi…

(btw my comment was/is about the DHT crawler)

You are describing a pay-to-play model. The validator is if the seeder/leech count is high. Well does DHT provide aggregate bandwidth of each torrent? If not, you can easily spin up 1000+ nodes and connect to your torrent. Tada fake popularity. If bandwidth is known, then you simply raise your costs a bit by running fake clients. There are anti-piracy groups who's entire mandate is to provide noise in the piracy ecosystem. Food for thought: bandwidth costs for this would be a rounding error for e.g. MGM, Universal, or any major content creator.

DHT does not offer any sort of reputation or comment system. Back to centralized torrenting which is why I suspect DHT crawling has not been a very popular feature

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#15

Post by no_time »

KoftaBob wrote: post #3
thelastparadise wrote: post #2

Very interesting. Does this approach actually work in practice? Also what happens if illegal content gets scooped up into the index?

Based on my understanding of how the torrent DHT works, all that’s happening is that youre requesting metadata on various torrent info hashes, but that’s not the same thing as actually downloading/seeding the content in the torrent itself.

>but that’s not the same thing as actually downloading/seeding the content in the torrent itself.

The question is whether Law Enforcement and "Intellectual Property" watchdogs make a meaningful distinction between the two in their monitoring tools.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#16

Post by danpalmer »

grepfru_it wrote: post #6
lobsterslive wrote: post #5

Earlier quoted context omitted.

It works well in practice. The DHT protocol includes announce messages that broadcast when new files are shared on BitTorrent. It then includes a "geometric" way to find people who are sharing those files. It doesn't include the files themselves, just the torrents which include a file list and location hashes. If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over ti…

>It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually h…

I don't think this project changes any of this? Torrents have been around for decades and this hasn't been a problem yet. We can't rule it out entirely but it does seem unlikely at this point to be worthwhile doing otherwise we'd see more exploitation.

If the criticism is that a DHT crawler is going to be more subject to this than a website where people submit upload torrents, that may be the case, but I think the author of this project underestimates the DHT crawling going on. I believe the torrent ecosystem is largely automated and there's little in the way of manual submission or human review going on.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#17

Post by [deleted] »

fluoridation wrote: post #8
grepfru_it wrote: post #6

Earlier quoted context omitted.

>It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually h…

The way to filter out garbage is to download things with lots of seeds, and if you still happen to download garbage, to immediately stop sharing it.

[deleted]

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#19

Post by justinclift »

no_time wrote: post #15
KoftaBob wrote: post #3

Earlier quoted context omitted.

Based on my understanding of how the torrent DHT works, all that’s happening is that youre requesting metadata on various torrent info hashes, but that’s not the same thing as actually downloading/seeding the content in the torrent itself.

>but that’s not the same thing as actually downloading/seeding the content in the torrent itself. The question is whether Law Enforcement and "Intellectual Property" watchdogs make a meaningful distinction between the two in their monitoring tools.

Seeing as they get so much other stuff wrong, it's highly likely they don't care.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#20

Post by grepfru_it »

SparkyMcUnicorn wrote: post #7
grepfru_it wrote: post #6

Earlier quoted context omitted.

>It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually h…

But you can still pick the option with the most seeders, which should get you what you're looking for most of the time. The spam problem isn't nonexistent within the centralized services either.

Hehe in a popular P2P client from the '03-'05 period, we said the same thing. Turns out there are groups with large amounts of funding which will provide a fake seed count. Either just faking metadata making it seem there was a high seed count but bogus nodes which would refuse connections (which was actual behavior from clients with bad ISPs - which we saw valid cases in asia or east europe) or would actually stream data (and some of them were on good hosts seeding multi mbps of bad data)

What i'm saying is it becomes a numbers game and those fake seeders usually have deep pockets financed by the content creators themselves

Post reply on HN