Updating Torrents via DHT Mutable Items • Hacker News Board

Re: Updating Torrents via DHT Mutable Items

#41

Post by aninhumer » Thu, Aug 11, 2016, 11:45 AM UTC

rakoo wrote: post #39

aninhumer wrote: post #31

Earlier quoted context omitted.

I remember someone setting up a bittorrent search engine based on DHT scraping. It's probably been taken down, but I think it was called DHT Dig, and there's apparently a library with that name. Something like that would make it easy to create centralised search engines for casual users, and allow power users to run local databases.

There was btdigg at some point ( https://btdigg.org/ ) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that: * In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/... , slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/... * In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf Basically all you have to d…

Ah yes, BTDigg was the name.

I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.

Re: Updating Torrents via DHT Mutable Items

#42

Post by aninhumer » Thu, Aug 11, 2016, 11:55 AM UTC

rakoo wrote: post #40

aninhumer wrote: post #30

Earlier quoted context omitted.

Indeed, I'm surprised it's taken this long for people to take advantage of bittorrent DHT for other p2p stuff. You don't even need to change the protocol to start using it. It's a ready made hash-to-peerlist mapping, so you could just use it for peer-discovery/bootstrapping and do other stuff out-of-band.

There already was a BEP to exchange arbitrary stuff, BEP44, for mutable or immutable items. It's taken this long to use this BEP specifically for mutable torrents, though, because it turned out that not many people had a need for mutable torrents.

Is this BEP44 actually implemented in most clients? I was experimenting with p2p some years ago, and I considered extending the protocol, but I figured it would be more practical to work out-of-band, rather than relying on widespread implementation of an extension (which didn't seem that likely to me at the time).

Re: Updating Torrents via DHT Mutable Items

#43

Post by niftich » Thu, Aug 11, 2016, 12:45 PM UTC

beefield wrote: post #33

niftich wrote: post #27

Earlier quoted context omitted.

Network effects and lack of off-the-shelf implementations.

Network effects are clearly a major issue, but wrapping the concept to a webpage and/or mobile app should not be a major hurdle. If I have not missed something. If I only had some more time on my hands...

Although the subject matter is different, GitTorrent [1] actually uses techniques you propose, by using BEP 44 to store Git revisions in the Bittorrent Mainline DHT, and Bitcoin to store cryptographically-signed usernames in the Bitcoin Blockchain.

You can look at how that implementation was done and what issues they encountered to see what it would take to implement a distributed [something], where that [something] in your case is a social network.

[1] https://github.com/cjb/GitTorrent

Re: Updating Torrents via DHT Mutable Items

#44

Post by niftich » Thu, Aug 11, 2016, 12:48 PM UTC

lkjhgfdsa57 wrote: post #28

niftich wrote: post #3

This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience. This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3]. [1] h…

Seems to fit the same usecase as Freenet USK keys: https://wiki.freenetproject.org/Updateable_Subspace_Key They allow the holder of the private key to add new data and clients who know the public USK can get notified of updates. In Freenet you can also get the old versions of the data for as long as they stay around on the network.

Freenet is remarkably well thought-out and solves a lot of these difficult issues like censorship resistance, deniable encryption, distributed data storage, and mutability; but it doesn't get much press these days.

Re: Updating Torrents via DHT Mutable Items

#45

Post by rakoo » Thu, Aug 11, 2016, 9:49 PM UTC

aninhumer wrote: post #41

rakoo wrote: post #39

Earlier quoted context omitted.

There was btdigg at some point ( https://btdigg.org/ ) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that: * In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/... , slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/... * In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf Basically all you have to d…

Ah yes, BTDigg was the name. I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.

Yep, that's the idea: if you join the DHT network with multpile, far away ids, you get more requests coming in. Unfortunately there is BEP 42 (http://www.bittorrent.org/beps/bep_0042.html) which restricts the number of IDs you can have on the same IP (the restriction is there for a good reason, so it's a good think to follow it).

Re: Updating Torrents via DHT Mutable Items

#46

Post by rakoo » Thu, Aug 11, 2016, 9:54 PM UTC

beefield wrote: post #26

So, what is stopping this to be the ultimate distributed social media platform? I mean, I should be able to publish a relatively simple sqlite database that contains: - my public posts - my public encryption key - list of my friends - my private posts encrypted with the friends' public keys Then I should seed at least my and my friends databases, so even if some of my friend is offline, his/hers database is still ava…

Twister (https://en.wikipedia.org/wiki/Twister_(software)) partly uses bittorrent, for long-term storage of posts and messages. It's not purely standard bittorrent, though.