Live data from Hacker News

Filecoin, StorJ and the problem with decentralized storage (2019)

randomoracle.wordpress.com

51–60 of 123 posts

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#51
Hi! CTO of Storj here. This article makes some reasonable points, and it makes some unreasonable ones.

First, on the reasonable: absolutely, privacy and security should be layered on top of whatever storage platform you use using your own encryption. No objection! That's the right approach. It is always better to bring your own encryption, and having it layered so your storage provider can't have access is great. I have much love for @cperciva's Tarsnap. It's awesome.

The downside with a product that punts encryption to the user is that users /typically don't do this/. In fact, users like to share things and serve content to others. Bringing your own encryption that the sharing infrastructure doesn't understand means it is much harder to actually use the product for sharing. This is why Storj includes hierarchical encryption for delegated sharing. It's not that embedding encryption into the framework is better encryption by any means, but it is a better default. Users are by default protected by their own keys that we don't have access to, which is a better default than the cloud. Should people use something else also? Sure! If it suits them.

On the unreasonable:

The author says:

> [I]t is very unlikely that a decentralized storage market can offer an alternative that can compete against centralized providers— AWS, Google, Azure— when measured on any of these [cost, speed, reliability] dimensions.

This is a testable assumption, and the author didn't test it (post is from December of 2019, we were in late beta then, working very well).

Storj is cheaper ($4/TB/mo, was $10/TB/mo in December of 2019), provides 11 9s of durability (no SLA offered until March 2020, but we had and still have never lost a single object out of billions), and is equivalent in speed to providers like Backblaze (and getting faster). It's also multiregion by default (obviously). This is a screaming deal on all dimensions. Try it out for yourself. Our metrics are delivering on all three dimensions.

This author is picking and choosing between architectures of Filecoin and Storj, which is fine, but each should be evaluated in isolation. Filecoin and Storj make very different design decisions with very different payoffs. The author argues Storj can't deliver on its promises with appeals to Filecoin's architecture.

Filecoin absolutely uses blockchains and proof of storage and whatever else. In fact, Filecoin does fall victim to requiring lots of resources (see https://docs.filecoin.io/mine/hardware-requirements/).

But this is just true of Filecoin, this isn't a problem with decentralized storage in general, and Storj is an excellent counter example. Storj does not require lots of resources (and in fact many of our node operators use Raspberry Pis). Storj is not a blockchain, uses statistical audits that are low-effort and low-CPU for storage node operators, and works great for idle capacity.

If you're interested in how we can achieve these things, you might take a look at an older blog post I wrote explaining (without naming) why Filecoin's architecture is fundamentally flawed: https://www.storj.io/blog/replication-is-bad-for-decentraliz...

Storj and Filecoin are both decentralized storage products, but they are really fundamentally very different, and their differences are worth understanding.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#52

I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated: 1. Economies of Scale 2. Network Properties (robustness, scalability, latency, etc.) It seems to me that a crucial element is missing from these analyses: organizational policy. All o…

a classic essay discussing the real ground truth problems of decentralized anything is by Jo Freeman, called the tyranny of structurelessness [0]. though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what th…

I'm familiar with this essay, and agree with most of its conclusions.

However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy!

I'm more interested in organizations that do have some degree of centralization. My hypothetical freelancers are certainly not disorganized. What I'm trying to grasp is the conditions under which any group -- especially one with centralized policymaking -- may seek decentralized infrastructure.

In other words: when does decentralized infrastructure serve the goals of a well-organized group?

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#53
As someone who's been interested in filecoin mining for a while, my main question is around legal liability for miners. For instance, if I am storing other people's data and it turns out someone was storing something illegal (unbeknownst to me), could I be held liable under US law?

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#54

Earlier quoted context omitted.

So granted the durability and availability is better than centralized solutions, you'd go with the distributed/decentralized solution? If that's the case, I think you could argue that open source and distributed solutions are much easier to make more reliable than centralized solutions. With a centralized solution, you're stuck with trusting their reliability or implement some reusable API (or use some 3rd party's AP…

I don't see how this addresses the reliability claim. There's a big difference between explaining that the data is stored in a Google cloud with certified 11-nines reliability, versus stored on a N number of machines in multiple clouds with unknown specs. How do I communicate the reliability of a distributed cloud that is essentially a collection of random servers? Is there a way that each of these server's reliabili…

You'd say something like "This content is hosted by 100 providers, with each having individual average uptime of 99.9%"

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#55

Earlier quoted context omitted.

a classic essay discussing the real ground truth problems of decentralized anything is by Jo Freeman, called the tyranny of structurelessness [0]. though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what th…

I'm familiar with this essay, and agree with most of its conclusions. However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy! I'm more interested in organizations that do have some degree of…

I believe such infrastructure is better thought of as distributed, rather than decentralized. decentralization is a tendency to resist centralizing forces, it is not an intentional policy of a group. as soon as an identifiable organ in the organization is charged with twiddling the policy knobs that govern control over group resource access, this organ is now distributing this access and it can no longer be considered decentralization.

the paradox is simply this: decentralization implies the absence of a decentralizer. there can be no "center of decentralization."

I contend that the term is much closer to incantation than implementation in practice.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#56

Earlier quoted context omitted.

So granted the durability and availability is better than centralized solutions, you'd go with the distributed/decentralized solution? If that's the case, I think you could argue that open source and distributed solutions are much easier to make more reliable than centralized solutions. With a centralized solution, you're stuck with trusting their reliability or implement some reusable API (or use some 3rd party's AP…

I don't see how this addresses the reliability claim. There's a big difference between explaining that the data is stored in a Google cloud with certified 11-nines reliability, versus stored on a N number of machines in multiple clouds with unknown specs. How do I communicate the reliability of a distributed cloud that is essentially a collection of random servers? Is there a way that each of these server's reliabili…

If you're willing to assume that each invididual server has some level of reliability (e.g. "one nine") and you assume failures are uncorrelated you can build a more reliable system using erasure coding and reason mathematically about the resulting reliability. http://www.suaybarslan.com/Reliability_Systems_14.pdf

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#57

I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated: 1. Economies of Scale 2. Network Properties (robustness, scalability, latency, etc.) It seems to me that a crucial element is missing from these analyses: organizational policy. All o…

a classic essay discussing the real ground truth problems of decentralized anything is by Jo Freeman, called the tyranny of structurelessness [0]. though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what th…

Structurelessness and decentralization are only related in that structureless things are decentralized. Decentralized things are rarely structureless. Whether a networking protocol is decentralized or not, it's definitely not structureless; it's nothing but structure, it's a protocol.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#58

Earlier quoted context omitted.

I'm familiar with this essay, and agree with most of its conclusions. However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy! I'm more interested in organizations that do have some degree of…

I believe such infrastructure is better thought of as distributed, rather than decentralized. decentralization is a tendency to resist centralizing forces, it is not an intentional policy of a group. as soon as an identifiable organ in the organization is charged with twiddling the policy knobs that govern control over group resource access, this organ is now distributing this access and it can no longer be considere…

> I believe such infrastructure is better thought of as distributed, rather than decentralized.

Hmm, I'm not sure I follow. On a technical level, a pure p2p system continues to function without central coordination even if each individual node is "owned" by the same entity. These seem to be two very different levels of analysis. If I understand your claim correctly, you are pointing out that this hypothetical entity is centralized -- and I agree -- but that doesn't magically turn the p2p system in question into something else.

To reformulate the question: when is it advantageous for this abstract entity to use a p2p system in this way rather than a client-server system?

Is it e.g. a question of scale?

I have a nagging suspicion that it's partly a question of usage-patterns, though I'm struggling to put my finger on it. This is what I mean by "organizational policy" (which is admittedly an opaque term...)

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#59

Earlier quoted context omitted.

a classic essay discussing the real ground truth problems of decentralized anything is by Jo Freeman, called the tyranny of structurelessness [0]. though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what th…

Structurelessness and decentralization are only related in that structureless things are decentralized. Decentralized things are rarely structureless. Whether a networking protocol is decentralized or not, it's definitely not structureless; it's nothing but structure, it's a protocol.

Well put. Thanks for putting words on that.

The confusion between structurelessness and decentralization is interesting in and of itself, though. It seems like it is exactly this conflation that prevents people from asking the interesting question: when does a centrally-organized social system (e.g. a business) benefit from a decentralized protocol?

It seems like this should have something to do with usage/access patterns within the business, yet we never hear about this.

Re: Filecoin, StorJ and the problem with decentralized storage (2019)

#60

As someone who's been interested in filecoin mining for a while, my main question is around legal liability for miners. For instance, if I am storing other people's data and it turns out someone was storing something illegal (unbeknownst to me), could I be held liable under US law?

This is where the DMCA and CDA safe harbors help you.
Post reply on HN