Crawling BitTorrent DHTs for Fun and Profit [pdf] • Hacker News Board

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#21

Post by fc417fc802 » Wed, Jun 24, 2026, 11:48 PM UTC

qingcharles wrote: post #6

NegativeLatency wrote: post #4

Earlier quoted context omitted.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

Are you running it at home? I built one with a nice TUI to run on a VPS so I can try and find rare magazine torrents, but Hetzner were upset about it. I need to find it a new home. It was a very good citizen, but it still raised too many flags.

I think generally you have to be very conservative about how you use ultra cheap hosts like hetzner simply due to the economics. Either find a more expensive service that will exert more effort towards discretion or alternatively spend $5 per month on a VPN that's friendly to torrents.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#22

Post by plusfour » Wed, Jun 24, 2026, 11:53 PM UTC

NegativeLatency wrote: post #4

hdgr wrote: post #3

Bitmagnet - https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

Same here, for over a year. how many torrents has yours indexed?

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#23

Post by muyuu » Thu, Jun 25, 2026, 12:17 AM UTC

hdgr wrote: post #3

Bitmagnet - https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

probably worth adding some ML filter to it because yeah, most of the bulky stuff in bittorrent is always going to be garbage - a lot like the internet generally, the value is in filtering the good stuff out

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#24

Post by the8472 » Thu, Jun 25, 2026, 12:37 AM UTC

Crawling has been somewhat simplified with BEP 51

https://bittorrent.org/beps/bep_0051.html

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#25

Post by farnsworthfusor » Thu, Jun 25, 2026, 12:46 AM UTC

fc417fc802 wrote: post #21

qingcharles wrote: post #6

Earlier quoted context omitted.

Are you running it at home? I built one with a nice TUI to run on a VPS so I can try and find rare magazine torrents, but Hetzner were upset about it. I need to find it a new home. It was a very good citizen, but it still raised too many flags.

I think generally you have to be very conservative about how you use ultra cheap hosts like hetzner simply due to the economics. Either find a more expensive service that will exert more effort towards discretion or alternatively spend $5 per month on a VPN that's friendly to torrents.

I heard the more money you're paying them the more lenient they are.

For cheap hosts look for ones that allow tor exit nodes if you're looking for ones that allow funny stuff. There are some that allow it for ideological reasons. Look through the hundreds on lowendtalk. On that forum you can even ask the providers directly if they allow it.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#26

Post by NoMoreNicksLeft » Thu, Jun 25, 2026, 12:48 AM UTC

qingcharles wrote: post #16

NoMoreNicksLeft wrote: post #10

Earlier quoted context omitted.

Which magazines?

Anything I don't have! Sometimes I'll find a torrent and no seeds/peers and I'll wonder if there is another torrent out there that has the same files in it somewhere that I can find. The other day it was trying to track down some older High Times issues that were torrented but the torrent is dead. Last night it was a mag titled Films & Filming which I know is scanned, but I can't find anywhere.

High Times is mostly on archive.org, if you need that one. I'd sort of like the film-making one, I'll put some time into that. On my list of periodicals, I think the count's up to 500 that I consider important enough to archive and I'm nowhere near done with it.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#27

Post by gonzalohm » Thu, Jun 25, 2026, 1:39 AM UTC

NegativeLatency wrote: post #4

hdgr wrote: post #3

Bitmagnet - https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

How much space do you need to store the index?

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#28

Post by infinite_spin » Thu, Jun 25, 2026, 2:29 AM UTC

muyuu wrote: post #23

hdgr wrote: post #3

Bitmagnet - https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

probably worth adding some ML filter to it because yeah, most of the bulky stuff in bittorrent is always going to be garbage - a lot like the internet generally, the value is in filtering the good stuff out

out of curiosity, what kind of junk/garbage is typical?

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#29

Post by plusfour » Thu, Jun 25, 2026, 3:26 AM UTC

gonzalohm wrote: post #27

NegativeLatency wrote: post #4

Earlier quoted context omitted.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

How much space do you need to store the index?

160gb for 27m torrents

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#30

Post by NegativeLatency » Thu, Jun 25, 2026, 4:11 AM UTC

qingcharles wrote: post #6

NegativeLatency wrote: post #4

Earlier quoted context omitted.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

Are you running it at home? I built one with a nice TUI to run on a VPS so I can try and find rare magazine torrents, but Hetzner were upset about it. I need to find it a new home. It was a very good citizen, but it still raised too many flags.

yes but with a VPN to my seedbox, all the egress from the docker containers its running in goes through this: https://github.com/passteque/gluetun

the seedbox is through https://www.feralhosting.com used them for over 10 years now and they've been great (shared hosting so I have linuxbrew setup there, but no docker sadly)