Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

bitmagnet.io

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#81

Post by chhs »

LeoPanthera wrote: post #47

> Pipe dream features > In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.

I wrote a project to do this using libtorrent a while ago, but unfortunately libtorrent crashes when seeding thousands of torrents at once, which is the current blocker. I haven't been working on it since.

https://github.com/chhs1/content-seeder

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#82

Post by chhs »

Scion9066 wrote: post #57
LeoPanthera wrote: post #47

> Pipe dream features > In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.

It might be able to do this with the Bittorent V2 format as the hashes would be per file.

V2 definitely makes it easier, but it's possible to identify files with V1 by iterating through metadata looking for files with the same length as a local file, then checking each piece against the local file. If they all match, then it's the same file. For boundary pieces where it consists of multiple files, I think it's safe enough to ignore if all of the remaining pieces match the local file, but you could do a more complex search looking for sets of files that all have the same length, and then compare the piece hash with the local files.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#83

Post by lost_tourist »

the8472 wrote: post #32
fallat wrote: post #9

It'd be so nice to have a super simple DHT crawler CLI tool, in both implementation and interface.

These things need uptime of hours and days to do it properly and also to stay up to date. There are millions of nodes and torrents and to be non-abusive you have to issue requests at a somewhat sedate pace. And activity kind of moves with the sun due to people who run torrent clients on their home machines. And there are lots of buggy or malicious implementations out there that you have to deal with. So you'd want to…

nah, that's tmux is for. I've had sessions running for a month+

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#84

Post by ShadowBanThis01 »

shepherdjerred wrote: post #62
complianceowl wrote: post #60

Guys, please educate me: I want to use torrents, but the thought of downloading something inappropriate by clicking on a deceptive link terrifies me (e.g., a download with the title of an action movie, but it turns out to be something else). How do you guys handle that risk?

If you're downloading "normal" stuff, then you aren't likely to run into an issue. Stick to reputable sites; reddit can help you figure out what those are.

What sites? Isn't the purpose of the software discussed here to make tracker sites unnecessary?

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#87

Post by drakenot »

> Something that looks like a decentralized private tracker; by this I probably mean something that’s based partly on personal trust and manually weeding out any bad actors; I’d be wary of creating something that looks a bit like Tribler, which while an interesting project seems to have demonstrated that implementing trust, reputation and privacy at the protocol level carries too much overhead to be a compelling alternative to plain old BitTorrent, for all its imperfections

I've thought about this problem a lot. Having a federated / distributed tracker but with some form of trust based, or opt-in curation would be amazing.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#88

Post by cchance »

pipes wrote: post #67
coolspot wrote: post #42

Please remove copyrighted movies from the screenshot on your website. It provides evidence that this program is designed for violating copyright, which makes DCMA takedown trivial.

And they probably want to remove text like "It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed."

There's nothing wrong with that since there are LOTS of free to share movies and tv shows especially those past their copyright dates.

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#89

Post by pepe56 »

Does this also share back DHT information like a "server"? I am not downloading torrents, but I always wished to have a daemon running on my servers to help the overall DHT networks health. Is there anything like this out there? A DHT server, that only collects, stores and gives back information?

Re: Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

#90

Post by no_time »

hattmall wrote: post #73
no_time wrote: post #12

Earlier quoted context omitted.

>If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over time), Correct me if I'm wrong but as far as I understand, passively listening on DHT would only mean you build up a list of infohashes of everything shared on BitTorrent. You'd actually have to reach out to your DHT peers to know what files the infohashes actually represents. Wrapping back to grandparent's ques…

Would this get hashes of items shared on private trackers too?

No because private trackers enforce that all torrents uploaded have DHT,PEX and LPD disabled. Usually done by a single tickbox that says “Make torrent private” in the client.

Of course, respecting these options in the torrent file is still up to the client. This is one of the reasons why all private trackers have a client whitelist too.

Post reply on HN