Updating Torrents via DHT Mutable Items
bittorrent.org
Updating Torrents via DHT Mutable Items
1–10 of 46 posts
Re: Updating Torrents via DHT Mutable Items
#2This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery.
Mutable Items are good for updating content, but that doesn't matter if nobody can find it.
There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this direction: http://www.bittorrent.org/beps/bep_0044.html although I don't think that it really goes far enough, as there is no standardization around how the data for searching will be structured, etc.
Re: Updating Torrents via DHT Mutable Items
#3This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience.
This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3].
[1] https://ipfs.io/
[2] http://www.bittorrent.org/beps/bep_0044.html
[3] https://gist.github.com/substack/eadd13302d785dc13aac#file-r...
Re: Updating Torrents via DHT Mutable Items
#4This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this directio…
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
Re: Updating Torrents via DHT Mutable Items
#5This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this directio…
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
An index? That could be huge. I think it's senseless.
So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?
Re: Updating Torrents via DHT Mutable Items
#6Earlier quoted context omitted.
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
An index? That could be huge. I think it's senseless. So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?
[deleted]
Re: Updating Torrents via DHT Mutable Items
#7This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this directio…
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
You don't see a problem with that..?
You still need to have the entry point - that's exactly where the censorship is occurring. Try to find a copy of the KickAssTorrents database on Google right now and tell me how easy it is.
The approach you're talking about is essentially what http://bitcannon.io/ is trying to do - but that requires running a separate app, a local MongoDB server, downloading many gigabytes of database data, and half an hour to import the torrent DB into your local instance, and that STILL requires you to be able to acquire the Torrent database or related infohash, which isn't easy because of the very censorship we're trying to sidestep.
Quite frankly, that's a really dumb solution to this censorship and it's naive to say that it's useful. This will never be used in the fashion you're describing and it does nothing for anti-censorship.
It makes so much more sense just to have the DHT be actually useful in surfacing content.
Re: Updating Torrents via DHT Mutable Items
#8Earlier quoted context omitted.
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
An index? That could be huge. I think it's senseless. So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?
Vuze (formerly known as Azureus) has (or at least had a prototype of) p2p search.
I wrote a paper on building a torrent search index quickly using the DHT records that this feature emitted at the time. [link redacted]
Re: Updating Torrents via DHT Mutable Items
#9Earlier quoted context omitted.
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
An index? That could be huge. I think it's senseless. So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?
The Mainline DHT (the more-supported DHT vs. Azureus/Vuze DHT) traditionally only contains (iirc, hashes of) torrent infohases mapped to peer lists. With BEP 44 (arbitrary data in DHT) or some other enhancement, this may change, but right now even if you traverse the DHT there's no "interesting" metadata about the keys (or, y'know, the actual content of the .torrent file) inside the DHT.
From the sibling comment (posted by swolchok)'s paper, a relevant quote:
"We chose to demonstrate our proof of concept only on Vuze due to the significant additional complexity of supporting two DHTs in the crawler. While Mainline does not contain torrent descriptions, its peer lists are keyed by the torrent infohashes themselves, so no additional machinery would be needed to discover infohashes. However, some method would be needed to discover torrent names, such as downloading .torrent files directly from peers using the metadata exchange protocol.
To explore this possibility, we built a prototype .torrent crawler. Since we are operating in the context of the Vuze DHT, our DHT crawler needs to obtain both a torrent description and a peer list before the .torrent crawler can contact the appropriate peers with the correct infohash. However, the .torrent files contain useful additional data, including the full listing of files contained in the torrent"
Re: Updating Torrents via DHT Mutable Items
#10This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience. This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3]. [1] h…
IPFS still doesn't have a content discovery mechanism, which I found very disappointing. It's not a killer app without that.