Live data from Hacker News

Crawling BitTorrent DHTs for Fun and Profit [pdf]

usenix.org

41–50 of 52 posts

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#41

Earlier quoted context omitted.

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

How much space do you need to store the index?

The factor that determines this most is how much of the files metadata for each torrent you choose to keep, which is configurable.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#42
post #3

Bitmagnet - https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

why the hell does that require a postgres database instance? has no one heard of sqlite? "It's just `docker compose up` dude." that's not excuse for unnecessary complexity. software for a single person should be a single binary that anyone can run how they choose. If they want to use Docker, great. If they want to run it in a terminal, great. If they want to run it on a server, great. If they want to run it on their…

The database gets quite large and the developers put in a lot of effort for tuning it for searchability.

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#44

The article neglects to define "DHT" before using it.

> The BitTorrent community has responded to these trends by deploying decentralizing tracking systems based on distributed hash tables (DHTs).

No it doesn't. Before you try to correct me, the abstract never counts ;)

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#46

The article neglects to define "DHT" before using it.

Distributed hash table - ButTorrent extension for discovering torrent's seeders by advertising its hash across known peer pool, think of it as a distributed tracker. Contrary to traditional way of asking a known tracker for peers of that torrent. Its algorithm is very elegant, using binary search on peers' and torrents' hashes, narrowing down to peers that are more likely to be seeders (or at least know some). https:…

Thanks!

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#48

The article neglects to define "DHT" before using it.

Downvote all you want, but a fact is a fact: The article doesn't define the abbreviation before using it.

I'm not sure why you're saying that in the HN comments. As far as I know the authors aren't there, and the paper is from 2010. It's not like it will get updated now.

One search tells you what DHT in the context of wikipedia is: https://en.wikipedia.org/wiki/Mainline_DHT

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#49
post #40

Earlier quoted context omitted.

I'm starting to upload everything very soon. I have >4million magazines here so far I think. Feel free to email me on my profile :)

Do you have old 2000s era Argos catalogues? Argos is a shop in the UK, I've been tracking down old catalogues to see if my childhood CD player is in there, can't find evidence of it anywhere and so many were made. The catalogues are sort of rare but were given away for free, now people sell old ones on ebay

I don't, but I've started a separate project to try and collect as many catalogues as possible, Argos included. I'm after a Maplin catalogue from my youth :) I do remember coming across a site with a bunch of Argos catalogues on it, but I suspect if you've done any searching you came across that site already?

Re: Crawling BitTorrent DHTs for Fun and Profit [pdf]

#50
post #23

Earlier quoted context omitted.

probably worth adding some ML filter to it because yeah, most of the bulky stuff in bittorrent is always going to be garbage - a lot like the internet generally, the value is in filtering the good stuff out

out of curiosity, what kind of junk/garbage is typical?

there's certainly massive amounts of porn, esp. if you look at active traffic, but perhaps even worse than that is random crap collected for no good reason and "backyard genius" trash content pooled together by the gigabyte

quality and traffic are grossly uncorrelated, so finding stuff is a huge challenge

Post reply on HN