Live data from Hacker News

Crawling BitTorrent DHTs for Fun and Profit [pdf]

usenix.org

31–40 of 52 posts

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#31

Earlier quoted context omitted.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

Same here, for over a year. how many torrents has yours indexed?

28M

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#32
post #23

Earlier quoted context omitted.

probably worth adding some ML filter to it because yeah, most of the bulky stuff in bittorrent is always going to be garbage - a lot like the internet generally, the value is in filtering the good stuff out

out of curiosity, what kind of junk/garbage is typical?

ungodly amounts of porn.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#33

Earlier quoted context omitted.

Anything I don't have! Sometimes I'll find a torrent and no seeds/peers and I'll wonder if there is another torrent out there that has the same files in it somewhere that I can find. The other day it was trying to track down some older High Times issues that were torrented but the torrent is dead. Last night it was a mag titled Films & Filming which I know is scanned, but I can't find anywhere.

Can I get a copy for the Internet Archive? Will take as much of the corpus as you’re willing to provide. (no affiliation with them)

I'm starting to upload everything very soon. I have >4million magazines here so far I think. Feel free to email me on my profile :)

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#34

Earlier quoted context omitted.

Anything I don't have! Sometimes I'll find a torrent and no seeds/peers and I'll wonder if there is another torrent out there that has the same files in it somewhere that I can find. The other day it was trying to track down some older High Times issues that were torrented but the torrent is dead. Last night it was a mag titled Films & Filming which I know is scanned, but I can't find anywhere.

High Times is mostly on archive.org, if you need that one. I'd sort of like the film-making one, I'll put some time into that. On my list of periodicals, I think the count's up to 500 that I consider important enough to archive and I'm nowhere near done with it.

Yeah, I have the High Times from archive.org, but it's missing a lot of issues which were torrented at some point. If there's anything you need send me an email on my profile.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#36

Earlier quoted context omitted.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

I disabled mine because it was constantly writing to my SSD.

In case you haven’t considered, HDDs work great for this, since write wear out isn’t really a factor for those. ZFS can also help batch together TXG’s until they hit time or size limits before commit it at once to the disk.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#39
post #3

Bitmagnet - https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

why the hell does that require a postgres database instance? has no one heard of sqlite?

"It's just `docker compose up` dude." that's not excuse for unnecessary complexity. software for a single person should be a single binary that anyone can run how they choose. If they want to use Docker, great. If they want to run it in a terminal, great. If they want to run it on a server, great. If they want to run it on their phone, great.

developers: avoid limiting people artificially like this. There is no way that Sqlite is insufficient for this.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#40

Earlier quoted context omitted.

Can I get a copy for the Internet Archive? Will take as much of the corpus as you’re willing to provide. (no affiliation with them)

I'm starting to upload everything very soon. I have >4million magazines here so far I think. Feel free to email me on my profile :)

Do you have old 2000s era Argos catalogues? Argos is a shop in the UK, I've been tracking down old catalogues to see if my childhood CD player is in there, can't find evidence of it anywhere and so many were made. The catalogues are sort of rare but were given away for free, now people sell old ones on ebay
Post reply on HN