Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Static torrent website with peer-to-peer queries over BitTorrent on 2M records

boredcaveman.xyz

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#91

Post by IshKebab »

pizza wrote: post #2

Yup. Calling it. This is the future, may not seem like it to everyone but this is a part of actually new and extremely useful, passively scalable technology. Imagine (good) unkillable zombie databases- so long as the name of a piece of data is known (its hash).. someone, somewhere, might make it possible for you to answer your query, without even needing to setup a server setup.. that’s _it_! Not to mention it might…

I don't think so. This still relies on non-distributed infrastructure to host the code and provide the database torrent. Those are not "unkillable". And an inefficient completely immutable database where even individual queries are not private does not sound like it has a lot of uses.

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#93

Post by beckman466 »

Aloha wrote: post #32

a decade ago, I had the idea of a distributed internet wide filesystem where the chunks could be duplicated over and over again over the internet. Something survivable and loosely/eventually consistent when updated. Someone appears to have built at least part of it.

have you seen rrDHT? https://blog.holochain.org/how-rrdht-works-a-tech-deep-dive/...

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#94

Post by reggieband »

octoberfranklin wrote: post #62
reggieband wrote: post #53

Earlier quoted context omitted.

This idea touches on (or rather, even expands on) an idea I've had brewing for a while. For those unfamiliar, there is a programming language called Unison [1] that has a nice feature where each function is identified by a hash of its AST. I often thought a cluster environment, maybe something BEAM-ish, would be an interesting idea where functions are retrievable based on a pair of (`func-name`, `hash`). You could ha…

This approach to mobile code in functional languages has been tried repeatedly since the 1990s. > each function is identified by a hash of its AST. That encoding for mobile code only works if the expression has no free variables (i.e. is closed). It turns out that it is surprisingly difficult to write code which you can be sure has no free variables at particular points (the "send this code to another machine for exe…

I agree a complete system capable of something as magical sounding as I described would be both extremely complex to implement and probably even a nightmare to program within.

As to the specific discussion of free variables and open/closed arguments, I have to admit I have no background or education in the theory behind programming languages. Other than going through about 50% of SICP about a decade ago, I also have very little experience with functional languages.

What has been fuelling this interest lately is learning a bit more about stack based languages like Forth. My extremely primitive understanding of such programming models suggests that a function/word in that kind of programming context is closed over some defined portion of the stack. So my naive mind considers that one could grab as much stack as necessary along with the word to be executed and just pipe that over to some other execution context. Of course, details matter and there are probably several important ones that I haven't even considered that would make this naive assumption border on impossible. However, it at least seems more reasonable to attempt than crawling through a heap trying to gather everything.

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#95

Post by octoberfranklin »

pharmakom wrote: post #84
octoberfranklin wrote: post #62

Earlier quoted context omitted.

This approach to mobile code in functional languages has been tried repeatedly since the 1990s. > each function is identified by a hash of its AST. That encoding for mobile code only works if the expression has no free variables (i.e. is closed). It turns out that it is surprisingly difficult to write code which you can be sure has no free variables at particular points (the "send this code to another machine for exe…

Would it be possible to copy any captured variables with the function? I’m sure smart people have figured out why this wouldn’t work.

How do you copy a variable? You can only copy values. A free variable has no value. You're confusing capture with closedness.

Here's a trivial example:

  \f -> \y -> (runRemotely (\x -> f y x))
The expression passed to runRemotely has a free variable "y". How are you going to serialize (\x -> f y x) in order to send it across the network? When you hit the "y", what are you going to do?

For this trivial minimalist example you might cook up a one-off hack like lambda-abstracting the free variables in the runRemotely expression and then reapplying to the returned value, but there are much more complicated and insidious examples where these workarounds don't work.

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#96

Post by octoberfranklin »

jlokier wrote: post #88
pharmakom wrote: post #84

Earlier quoted context omitted.

Would it be possible to copy any captured variables with the function? I’m sure smart people have figured out why this wouldn’t work.

In pure functional programs it's possible to copy the state (monad/environment/free variables), but it's not always efficient, depending on what those are. In general programs, doesn't have to be functional, the environment is stateful and often has abstract, black box processes. This can be transferred as well, by copying some things, transforming others, and in general where necessary using a two-way pipe of some k…

It's not a question of efficiency. You can copy values but you can't copy unbound free variables.

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#97

Post by chatmasta »

remram wrote: post #50
chatmasta wrote: post #31

Inspired by this, can anyone explain why distributed protocols more often opt for centralized consensus algorithms like Raft, instead of decentralized schemes like Chord or Kademlia? In all cases, the underlying data structure is a shared key/value store. Intuitively, the p2p approach feels more robust, since each node only needs to worry about itself, and every node is the same. So why add the coordinator node? Is i…

Obviously you don't need consensus protocols if you are not trying to build consensus... It's like asking why we need filesystems and network cards if all we're trying to do is show lights on screens. Obviously not all patterns of lights are as easy to produce on computers.

I'm not talking about the consensus protocol of the blockchain itself, but of the p2p algorithms underlying it, e.g. using Kademlia for service discovery and message routing. I'm asking why a distributed system would choose something like Consul (which uses Raft, and requires a coordinator node) instead of running a decentralized protocol like Kademlia (which has no coordinator nodes) within their distributed single-tenant environment.

I did a bit more research last night, and discovered that Bitfinex actually does something like this internally (anyone know if this is up to date?) [0] — they built a service discovery mesh by storing arbitrary data on a DHT implementing BEP44 (using webtorrent/bittorrent-dht [1]).

This seems pretty cool to me, and IMO any modern distributed system should consider running decentralized protocols to benefit from their robustness properties. Deploying a node to a decentralized protocol requires no coordination or orchestration, aside from it simply joining the network. Scaling a service is as simple as joining a node to the network and announcing its availability as an implementation of that service.

At first glance, this looks like a competitive advantage, because it decouples the operational and maintenance costs of the network from its size.

So I'm wondering if there is a consistent tradeoff in exchange for this robustness — are decentralized applications more complex to implement but simpler to operate? Is latency of decentralized protocols (e.g. average number of hops to lookup item in a DHT) untenably higher than that of distributed protocols (e.g. one hop once to get instructions from coordinator, then one hop to lookup item in distributed KV)? Does a central coordinator eliminate some kind of principle agent problem, resulting in e.g. a more balanced usage of the hashing keyspace?

Decentralization emerged because distributed solutions fail in untrusted environments — but this doesn't mean that decentralized solutions fail in trusted environments. So why not consider more decentralized protocols to scale internal systems?

[0] https://github.com/bitfinexcom/grenache

[1] https://github.com/webtorrent/bittorrent-dht

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#98

Post by remram »

chatmasta wrote: post #97
remram wrote: post #50

Earlier quoted context omitted.

Obviously you don't need consensus protocols if you are not trying to build consensus... It's like asking why we need filesystems and network cards if all we're trying to do is show lights on screens. Obviously not all patterns of lights are as easy to produce on computers.

I'm not talking about the consensus protocol of the blockchain itself, but of the p2p algorithms underlying it, e.g. using Kademlia for service discovery and message routing. I'm asking why a distributed system would choose something like Consul (which uses Raft, and requires a coordinator node) instead of running a decentralized protocol like Kademlia (which has no coordinator nodes) within their distributed single-…

Raft is a consensus protocol. You don't need it if you don't need consensus.

You can do service discovery etc with gossip protocols. You are right that you don't need consensus to have systems publish their own keys on a network.

Systems that use Raft or equivalent do have a need for consensus, for example CassandraDB/CockroachDB (when you do `UPDATE account SET balance = balance - 10 WHERE user='chatmasta';`, you need that transaction to go through only once globally, and you need the whole system to agree on whether it did), Kubernetes (when you ask for a database to be served on some hostname, you need a single instance to go up, and every load balancer to route to that same instance), etc.

If you have examples of systems that use strong consensus when it's not required, point them out. Stating "why do some systems use strong consensus when other systems (doing something completely different) get by without consensus" is a bit strange.

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#99

Post by sliken »

pizza wrote: post #2

Yup. Calling it. This is the future, may not seem like it to everyone but this is a part of actually new and extremely useful, passively scalable technology. Imagine (good) unkillable zombie databases- so long as the name of a piece of data is known (its hash).. someone, somewhere, might make it possible for you to answer your query, without even needing to setup a server setup.. that’s _it_! Not to mention it might…

First CAS (content addressable storage) is nothing new. Torrents aren't new and are basically a distribed CAS database. With a magnet link (basically a checksum) you can find those files hosted by any torrent client on the planet.

So sure things are "unkillable", but depend on someone/somewhere decided it's worth storing that particular chunk. Seems strange to pair "unkillable" which depends on someone somewhere that "might" do something.

Much like how IPFS was easily oversold, great you can find things hosted anywhere on the planet, but if you publish 1M files and expect to magically seem them hosted elsewhere a year later you are likely to be disappointed.

I do wonder if it would be a better approach to replace filecoin or similar complex trust relationships with a simple peer to peer trading program. Something along the lines of "Lets trade 128MB", trust but verify, then watch uptime/availability. For clients up less than a month, give them the free 128MB and watch, 1-3 months trust them enough to store data with a 20x replication, 3-12 months 10x replication, over a year 5x, whitelisted peers of friends/family 3x.

Re: Static torrent website with peer-to-peer queries over BitTorrent on 2M records

#100

Post by pharmakom »

octoberfranklin wrote: post #95
pharmakom wrote: post #84

Earlier quoted context omitted.

Would it be possible to copy any captured variables with the function? I’m sure smart people have figured out why this wouldn’t work.

How do you copy a variable? You can only copy values. A free variable has no value. You're confusing capture with closedness. Here's a trivial example: \f -> \y -> (runRemotely (\x -> f y x)) The expression passed to runRemotely has a free variable "y". How are you going to serialize (\x -> f y x) in order to send it across the network? When you hit the "y", what are you going to do? For this trivial minimalist examp…

I was talking in the context of pure functional programming. You can send the AST of a lambda.
Post reply on HN