Filecoin, StorJ and the problem with decentralized storage (2019) • Hacker News Board

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#81

Post by krrrh » Thu, Apr 22, 2021, 10:51 PM UTC

omginternets wrote: post #43

omginternets wrote: post #28

I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated: 1. Economies of Scale 2. Network Properties (robustness, scalability, latency, etc.) It seems to me that a crucial element is missing from these analyses: organizational policy. All o…

Addendum: I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0]. Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible…

This speaks to the disingenuity of a lot of the decentralization talk in the crypto space. The Beaker browser and the hyperdrive protocol (formerly DAT) is an interesting case study. It has a level of maturity and delivers on many of the ideas that helped filecoin and other token-financed projects raise hundreds of millions of dollars, but it is relatively obscure and I don’t think I know anyone who has used it other than the people I’ve encouraged to check it out. If there were actual demand for dweb technology then it and other projects like it should be getting a lot more attention since it’s a free and relatively user friendly platform compared to services that require managing token exchange risk, accounting, and hassle.

https://beakerbrowser.com/

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#82

Post by brutaltruth » Thu, Apr 22, 2021, 11:13 PM UTC

krrrh wrote: post #79

300bps wrote: post #63

Earlier quoted context omitted.

What do you base AWS storage having "an enormous markup" on? AWS has a type of storage to meet any access need / budget. S3 Glacier Deep Dive for example is perfect for storing archival items and costs $0.00099 per GB-month. So you can store a TB for about $1 per month. There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost…

It’s like the people who seriously touted Bitcoin as a cheaper payment mechanism in than credit cards 2013. Few of them explored how ~3% fees are a balance between customer rewards, regulatory priorities, and merchant convenience. In places like Australia regulators made other choices, largely eliminating rewards programs and capping interchange fees at around a tenth of what they are in the US. There are good argume…

Clearly the Bitcoin skeptics had the better of the debate ;)

You must also think "scale" and "market pressures" keep down the price of Windows.

Hint: AWS has wonderful margins because everything is tied together, and bandwidth charges to get data out are huge. S3 is a great solution for data used on AWS, but costs more than 30x Storj on bandwidth alone ($90/TB for outside access, plus the cost of doing cross-region replication).

If you're going to work from "first principles" it always helps to have the first hint about how markets actually work--and that different products make different trade-offs (local transfer speed and latency vs. cheap global redundancy)

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#83

Post by ruggeri » Thu, Apr 22, 2021, 11:22 PM UTC

The article raises a question I've wondered about before: how do you know that anyone will transfer you your data when you need it? You can require people to prove that they stored it, but how can you require people to transfer it out to you?

You presumably agree with the node to pay for the transfer out, but if both counterparties are anonymous and do not trust each other, how is this accomplished?

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#84

Post by px43 » Thu, Apr 22, 2021, 11:27 PM UTC

nemo1618 wrote: post #6

yosito wrote: post #2

As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least out of the box, zero-knowledge encryption was easier, though I know there are ways to achieve zero knowledge encryption with any S3 provider. I hadn't considered Glacier, but I may look into it. Edit: One thing I liked about Storj was that geographic redundan…

Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer. Instead, decentralized storage gives you the ability to store your data with dozens…

> SLAs -- something that a blockchain can't really offer.

Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely.

If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties.

The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#85

Post by dane-pgp » Thu, Apr 22, 2021, 11:29 PM UTC

JoshuaDavid wrote: post #77

omginternets wrote: post #59

Earlier quoted context omitted.

Well put. Thanks for putting words on that. The confusion between structurelessness and decentralization is interesting in and of itself, though. It seems like it is exactly this conflation that prevents people from asking the interesting question: when does a centrally-organized social system (e.g. a business) benefit from a decentralized protocol? It seems like this should have something to do with usage/access pat…

You can have structure without centralization - see for example federated systems like email. However, the lack of a centralized authority comes with its own issues - see for example federated systems like email.

I won't disagree with your summary of the situation with email, but I think I would add that it is possible to have federated reputation systems which would solve much of the spam problem with email, at least.

To give more detail about the sort of system I mean, here is an excellent description of the approach favoured by the team behind the Matrix communication protocol:

https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix...

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#86

Post by lxdesk » Thu, Apr 22, 2021, 11:33 PM UTC

omginternets wrote: post #59

pessimizer wrote: post #57

Earlier quoted context omitted.

Structurelessness and decentralization are only related in that structureless things are decentralized. Decentralized things are rarely structureless. Whether a networking protocol is decentralized or not, it's definitely not structureless; it's nothing but structure, it's a protocol.

Well put. Thanks for putting words on that. The confusion between structurelessness and decentralization is interesting in and of itself, though. It seems like it is exactly this conflation that prevents people from asking the interesting question: when does a centrally-organized social system (e.g. a business) benefit from a decentralized protocol? It seems like this should have something to do with usage/access pat…

Here's an example: Cash decentralizes credit.

"Cash and carry" grocery outlets were a 20th century innovation. [0] Before that, the norm was to have a line of credit with the business, and in many cases, to accept their terms for delivery. Cash transactions anonymize, since settlement is done at the counter. You don't have to assess the buyer, you just need to verify the bills and change are real.

However, that isn't the entire story. While there were businesses using cash before this, they faced difficulties with accounting, supply logistics, and other elements that made it hard to conceive of something like a "supermarket", carrying a vast variety and quantity of goods on a daily basis. So then we have to look at all the pieces that fell into place to make it possible.

The automobile decentralized access to mobility, making carry-away a real possibility for more people, and thus making it possible to apply cash-and-carry in more places. Supporting elements like cash registers and refrigeration were becoming mature enough to support new forms of retail and allow more parts of the transaction to be delegated to local outlets and low-wage employees. The inter-war years really saw a whole set of technological innovations that were used in combinations to propel social changes and different categories of business(e.g. fast food), many of them decentralized in some respects but centralized in others - supermarket chains, as opposed to local markets.

These are the kinds of changes that are hardest to assess in full; when you decentralize one thing, centralization is "squeezed" into other parts of the economy, it seems. The obvious example for this phenomenon is Amazon, leveraging an apparently decentralizing mechanism(online retail - premised on an internet with sufficient bandwidth and security to list goods and take payments) into becoming the world's largest retailer. So it's centralized on one axis, but decentralized on others - a buyer no longer has to go to a particular physical location to purchase something, when all of it can be delivered to the doorstep.

[0] https://en.wikipedia.org/wiki/Cash_and_carry_(wholesale)

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#87

Post by boogoob » Thu, Apr 22, 2021, 11:55 PM UTC

brutaltruth wrote: post #46

Not very interesting. There are ways to solve all of these concerns--no fees are paid during the first X months, nodes periodically tested for bandwidth, etc. etc. Just because the author didn't bother to think it through doesn't mean it's not straightforward. The whole idea of a blockchain is reputation CAN be built up over time, because every transaction is recorded and there might even be incentives to associate n…

I agree that all the concerns can be mitigated. But they're not just theoretical, they're also issues with the current implementations—some of which remain a year and a half after this was written.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#88

Post by wmf » Fri, Apr 23, 2021, 12:02 AM UTC

px43 wrote: post #84

nemo1618 wrote: post #6

Earlier quoted context omitted.

Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer. Instead, decentralized storage gives you the ability to store your data with dozens…

> SLAs -- something that a blockchain can't really offer. Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then thei…

I think this speaks to the difference between what SLAs really are and what people wishfully think they are. People think an uptime SLA means "we will not go down much" but it really means "one day we will go down hard and we'll give you a refund (which will be insignificantly small compared to the pain we caused you)".

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#89

Post by benlivengood » Fri, Apr 23, 2021, 12:03 AM UTC

capableweb wrote: post #45

wmf wrote: post #38

Earlier quoted context omitted.

Given that public clouds are ruthlessly optimized for efficiency and they don't have to perform trustless proof of anything, it's virtually certain that decentralized storage is less efficient in every way than centralized.

What does "virtually certain" mean here anyway? And on what scale are you imagining things? Let's say the entirety of YouTube was decentralized in a way that every internet connected device plugged in to the wall starts sharing watched content to other viewers who want to watch that content. If it's your neighbor, they'll download it straight from you, via the closest router/hub. Suddenly the company's hosting is mor…

Youtube (and most other streaming services) is already decentralized into thousands of edge caches. Decentralizing it further doesn't make much sense because the network cost is about the same if individual homes or their nearest ISP DC has the cache.

Most home networks are wireless only at this point which is pretty bad for throughput, latency, and reliability if serving other homes. Maybe ISPs could start building some storage and IPFS into modems? I think handling the metadata of a p2p system might still be too costly. Imagine every neighborhood having to run a torrent tracker for every media file produced in the last ~year.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#90

Post by wmf » Fri, Apr 23, 2021, 12:06 AM UTC

ruggeri wrote: post #83

The article raises a question I've wondered about before: how do you know that anyone will transfer you your data when you need it? You can require people to prove that they stored it, but how can you require people to transfer it out to you? You presumably agree with the node to pay for the transfer out, but if both counterparties are anonymous and do not trust each other, how is this accomplished?

The answer is incentives: if you're paying to retrieve data, why shouldn't they take your business?