Live data from Hacker News

Updating Torrents via DHT Mutable Items

bittorrent.org

11–20 of 46 posts

Re: Updating Torrents via DHT Mutable Items

#11
post #8
post #5

Earlier quoted context omitted.

An index? That could be huge. I think it's senseless. So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?

Vuze (formerly known as Azureus) has (or at least had a prototype of) p2p search. I wrote a paper on building a torrent search index quickly using the DHT records that this feature emitted at the time. [link redacted]

Great work with this. Are you still in academia doing research?

Re: Updating Torrents via DHT Mutable Items

#12
post #10
post #3

This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience. This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3]. [1] h…

IPFS still doesn't have a content discovery mechanism, which I found very disappointing. It's not a killer app without that.

Well, to be fair, neither does HTTP, short of either name-services (DNS), or crawling-and-indexing.

IFPS has IPNS [1], the name services (the feature most-directly relevant to the Bittorrent enhancement in question).

To date, I'm unaware of an IFPS crawler-and-indexer search engine.

[1] https://github.com/ipfs/faq/issues/16

Re: Updating Torrents via DHT Mutable Items

#13
post #12
post #10

Earlier quoted context omitted.

IPFS still doesn't have a content discovery mechanism, which I found very disappointing. It's not a killer app without that.

Well, to be fair, neither does HTTP, short of either name-services (DNS), or crawling-and-indexing. IFPS has IPNS [1], the name services (the feature most-directly relevant to the Bittorrent enhancement in question). To date, I'm unaware of an IFPS crawler-and-indexer search engine. [1] https://github.com/ipfs/faq/issues/16

Why is that fair? They're designed for different purposes. IPFS falls far short of being useful in a transformative way because it lacks this important feature. Otherwise, it'll be a kind of gimmicky alternative to HTTP, "auto-archiving HTTP", that never got used for anything real.

Re: Updating Torrents via DHT Mutable Items

#14
post #13
post #12

Earlier quoted context omitted.

Well, to be fair, neither does HTTP, short of either name-services (DNS), or crawling-and-indexing. IFPS has IPNS [1], the name services (the feature most-directly relevant to the Bittorrent enhancement in question). To date, I'm unaware of an IFPS crawler-and-indexer search engine. [1] https://github.com/ipfs/faq/issues/16

Why is that fair? They're designed for different purposes. IPFS falls far short of being useful in a transformative way because it lacks this important feature. Otherwise, it'll be a kind of gimmicky alternative to HTTP, "auto-archiving HTTP", that never got used for anything real.

I don't disagree that having content discovery would be nice, but IFPS makes a pretty good case [1] about what it brings to the table over HTTP. My point was that HTTP also has "zero out-of-the-box support" for content discovery, yet once search engines came out, we were fine.

[1] https://ipfs.io/#why

Re: Updating Torrents via DHT Mutable Items

#15
post #2

This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this directio…

Tribler is a research project of Delft University of Technology. (...) Tribler is the first client which continuously improves upon the aging BitTorrent protocol from 2001 and addresses its flaws. We expanded it with, amongst others, streaming from magnet links, keyword search for content, channels and reputation-management. All these features are implemented in a completely distributed manner, not relying on any centralized component. Still, Tribler manages to remain fully backwards compatible with BitTorrent.

https://www.tribler.org/about.html

Re: Updating Torrents via DHT Mutable Items

#16
post #10
post #3

This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience. This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3]. [1] h…

IPFS still doesn't have a content discovery mechanism, which I found very disappointing. It's not a killer app without that.

I don't really believe the same network can carry its metadata. (attached directly to the content) I mean, in an ideal world it would be great, but in practice how would that work? You can copy anyone's metadata to the file you published. How can anyone tell a difference? (unless you put some kind of signed ranking system on top of that, but that's just another layer waiting to be abused - like kazaa's star ratings)

It's basically the reason search engines stopped trusting meta tags on the websites.

Re: Updating Torrents via DHT Mutable Items

#17
post #13
post #12

Earlier quoted context omitted.

Well, to be fair, neither does HTTP, short of either name-services (DNS), or crawling-and-indexing. IFPS has IPNS [1], the name services (the feature most-directly relevant to the Bittorrent enhancement in question). To date, I'm unaware of an IFPS crawler-and-indexer search engine. [1] https://github.com/ipfs/faq/issues/16

Why is that fair? They're designed for different purposes. IPFS falls far short of being useful in a transformative way because it lacks this important feature. Otherwise, it'll be a kind of gimmicky alternative to HTTP, "auto-archiving HTTP", that never got used for anything real.

Did you ever use the web in 1995?

Content discovery is neither a part of HTTP nor the architecture of the web. It's a feature if it's current landscape, built on top of existing non-ideal ALI and brute force (scanning).

Good web search (represented by Google) appeared around 1998, years after the general availability of the web, when the corpus of web pages was already large. Up to this moment, search is powered by ad revenue.

I don't see how IPFS is significantly different in this regard.

Re: Updating Torrents via DHT Mutable Items

#19
post #17
post #13

Earlier quoted context omitted.

Why is that fair? They're designed for different purposes. IPFS falls far short of being useful in a transformative way because it lacks this important feature. Otherwise, it'll be a kind of gimmicky alternative to HTTP, "auto-archiving HTTP", that never got used for anything real.

Did you ever use the web in 1995? Content discovery is neither a part of HTTP nor the architecture of the web. It's a feature if it's current landscape, built on top of existing non-ideal ALI and brute force (scanning). Good web search (represented by Google) appeared around 1998, years after the general availability of the web, when the corpus of web pages was already large. Up to this moment, search is powered by a…

It isn't 1995, this is exactly my point. If we're trying to design new systems, we shouldn't design them with the exact same problems. What's the point of "decentralizing" if it just means another Google?

Re: Updating Torrents via DHT Mutable Items

#20
post #2

This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this directio…

Tribler is a research project of Delft University of Technology. (...) Tribler is the first client which continuously improves upon the aging BitTorrent protocol from 2001 and addresses its flaws. We expanded it with, amongst others, streaming from magnet links, keyword search for content, channels and reputation-management. All these features are implemented in a completely distributed manner, not relying on any cen…

I worked on an anonymous P2P research project many many years ago now, and we were shocked when we saw how much funding they received compared to us (3 mil EUR!) - but I'm super happy to see that their project is still alive and kicking! I guess the money was put to good use.
Post reply on HN