Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magical Block Store: Why EBS Can't Work

joyeur.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Magical Block Store: Why EBS Can't Work

#31

Post by jamesaguilar »

sigil wrote: post #30
jamesaguilar wrote: post #29

Earlier quoted context omitted.

I doubt it. Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect. Although networks will never be perfectly reliable, nor machines, they seem to be getting more reliable. Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are…

> Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect. You mean, in that case tolerance to partition and availability should be perfect. > Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are lower than the odds of you get…

> You mean, in that case tolerance to partition and availability should be perfect.

No. If a network is never partitioned, you don't need to write algorithms that can tolerate partitions. Therefore consistency and availability are possible.

> So this is the really interesting question. All the CAP theorem says is that (C,A,P) != (1.0,1.0,1.0). How close to (1.0,1.0,1.0) could we make (C,A,P)? If infinitely close, then we have achieved perfection by the limit, and the CAP theorem is rather pointless. If not, then what is the numeric limit?

I think you have misunderstood the theorem (at least, if my bachelor-degree-level understanding is correct). C, A, and P are not variables you can multiply together or perform mathematical operations on. They are more like booleans. "Is the web service consistent (are requests made against it atomically successful or unsuccessful)?" "Is the web service available (will all requests to it terminate)?" "Is the web service partition-tolerant (will the other properties still hold if some nodes in the system cannot communicate with others)?" These questions cannot be "0.5 yes". They are either all-the-way-yes or all-the-way-no.

> . . . and the CAP theorem is rather pointless

Not really. It is pointful for networks that experience partitions. It just doesn't apply to reliable networks. It also sort-of doesn't apply when an unreliable network is acting reliably, with the caveat that since it is not possible to tell in advance when a network will stop behaving reliably, you still have to choose between these three properties when writing your algorithms for when the network behaves badly.

Re: Magical Block Store: Why EBS Can't Work

#32

Post by spullara »

jinushaun wrote: post #24
spullara wrote: post #4

Also worth noting is that Amazon isn't forcing you to use EBS. They also have tons of fast local storage available to RAID as you wish.

Except Amazon does... Their free micro instance only supports EBS. For some of us who just want to dip our toes into AWS without too much money or commitment, EBS is our only choice.

Then you really don't need to worry about fault tolerance or uptime.

Re: Magical Block Store: Why EBS Can't Work

#33

Post by johnb »

sciurus wrote: post #27
johnb wrote: post #19

It's funny how disk abstractions get you every time. We used to store and process all of our uploads from our rails app on a GFS partition. GFS behaved like a normal disk most of the time, but we started having trouble processing concurrent uploads and couldn't replicate in dev. It turned out so GFS could work at all, it had different locking than regular disks. Every time you created a new file it had to lock the co…

FWIW, this behavior is explained early on in the documentation for GFS2.

As we were using EngineYard for hosting at the time, everything was set up for us and we never thought to look it up.

We now pay a lot more attention to underlying stack. Just because you've outsourced hosting (either cloud or managed physical servers), you really need to know every component yourself.

Re: Magical Block Store: Why EBS Can't Work

#34

Post by sigil »

jamesaguilar wrote: post #31
sigil wrote: post #30

Earlier quoted context omitted.

> Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect. You mean, in that case tolerance to partition and availability should be perfect. > Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are lower than the odds of you get…

> You mean, in that case tolerance to partition and availability should be perfect. No. If a network is never partitioned, you don't need to write algorithms that can tolerate partitions. Therefore consistency and availability are possible. > So this is the really interesting question. All the CAP theorem says is that (C,A,P) != (1.0,1.0,1.0). How close to (1.0,1.0,1.0) could we make (C,A,P)? If infinitely close, the…

> C, A, and P are not variables you can multiply together or perform mathematical operations on. They are more like booleans.

Right, but I wasn't restating CAP, just wondering about a follow on to CAP that considers the probability of remaining consistent, the probability of remaining available, and the probability of no failures due to network partitions in physical terms.

Is this not an interesting thing to consider? What if someone proves a hard limit on the product of these probabilities in some physical computation context? The CAP theorem is absolutely fascinating to me, especially if it has something real to say about the systems we can build in the future. The future looks even more distributed.

> It is pointful for networks that experience partitions. It just doesn't apply to reliable networks.

Is there such a thing as a "reliable" network when thousands or millions of computational nodes are involved? Are the routers and switches which connect such a network 100% available? If an amplification attack saturates some network segment with noise, what then?

As programmers, we desperately want things to work, and it's easy to greet something like CAP with flat out denial. I know I'm always fighting it. "It will never fail." No, it can and will fail.

Re: Magical Block Store: Why EBS Can't Work

#35

Post by jamesaguilar »

sigil wrote: post #34
jamesaguilar wrote: post #31

Earlier quoted context omitted.

> You mean, in that case tolerance to partition and availability should be perfect. No. If a network is never partitioned, you don't need to write algorithms that can tolerate partitions. Therefore consistency and availability are possible. > So this is the really interesting question. All the CAP theorem says is that (C,A,P) != (1.0,1.0,1.0). How close to (1.0,1.0,1.0) could we make (C,A,P)? If infinitely close, the…

> C, A, and P are not variables you can multiply together or perform mathematical operations on. They are more like booleans. Right, but I wasn't restating CAP, just wondering about a follow on to CAP that considers the probability of remaining consistent , the probability of remaining available , and the probability of no failures due to network partitions in physical terms. Is this not an interesting thing to consi…

I still don't understand what you mean when you say, "probability of remaining consistent," etc. Either you wrote the service so it system would always remain consistent or you didn't. Similarly with availability. Either the system will always return a result, or it may sometimes hang.

Maybe what you mean is the probability of whichever of C, A, or P you gave up actually becoming a problem? But I cannot imagine a physical law of the form you are referring to applying uniformly to these disparate properties. I wouldn't even know how to formulate it for consistency. For availability and partition tolerance it would just be, "Requests to this service will (availability: hang forever/partition-tolerance: return with errors) at a rate exactly equal to the probability of network failures."

With regards to your last point, there are no reliable networks, at least where I work. That doesn't mean there won't be.

Re: Magical Block Store: Why EBS Can't Work

#36

Post by arethuza »

sigil wrote: post #23
blantonl wrote: post #15

Earlier quoted context omitted.

Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it. That presumption assumes that the application is being used as the right tool to resolve the problem. And it also assumes that "the problem" is a finite and solvable item.

> And it also assumes that "the problem" is a finite and solvable item. Yes. To make this a bit more concrete, if "the problem" is making distributed storage look and behave exactly like local storage, the CAP Theorem has something to say about its solvability.

Out of interest, where do you draw the line between local storage and distributed storage? By local storage do you mean directly a attached storage?

What about FC SANs or iSCSI over a WAN? Are they local or distributed?

Re: Magical Block Store: Why EBS Can't Work

#37

Post by agazso »

edw wrote: post #21

This awesome entry perfectly captures why I have always hated NFS. I can deal with the possibility that if a machine's hard drive dies, my system is going to have a very hard time continuing to operate in a normal manner, but then NFS comes along, and you realize that all sorts of I/O operations that previously employed a piece of equipment that failed once every two and a half years now depend on a working network w…

I hate NFS for the same reasons that you just wrote, but to be honest it is not a protocol issue, but rather an implementation issue.

If NFS were implemented totally in userspace (like FTP), it would not hang the entire system when something breaks. On the other hand, it would be much slower than it is, therefore it would be unsuitable for a lot of use-cases where it is used now.

Re: Magical Block Store: Why EBS Can't Work

#38

Post by CPlatypus »

He only gets it half right. A filesystem interface instead of a block interface is the right choice IMO. Private storage instead of distributed storage is the wrong choice for capacity, performance, and (most importantly) availability reasons. They didn't go with a ZFS-based solution because it was the best fit to requirements. They went with it because they had ZFS experts and advocates on staff.

As Schopenhauer said, every man mistakes the limits of his own vision for the limits of the world, and these are people who've failed to Get It when it comes to distributed storage ever since they tried and failed to make ZFS distributed (leading to the enlistment of the Lustre crew who have also largely failed at the same task). If they can't solve a problem they're arrogant enough to believe nobody can, so they position DAS and SAN as the only possible alternatives.

Disclaimers: I'm the project lead for CloudFS, which is IMO exactly the kind of distributed storage people should be using for this sort of thing. I've also had some fairly public disputes with Bryan "Jackass" Cantrill, formerly of Sun and now of Joyent, about ZFS FUD.

Re: Magical Block Store: Why EBS Can't Work

#39

Post by moe »

agazso wrote: post #37
edw wrote: post #21

This awesome entry perfectly captures why I have always hated NFS. I can deal with the possibility that if a machine's hard drive dies, my system is going to have a very hard time continuing to operate in a normal manner, but then NFS comes along, and you realize that all sorts of I/O operations that previously employed a piece of equipment that failed once every two and a half years now depend on a working network w…

I hate NFS for the same reasons that you just wrote, but to be honest it is not a protocol issue, but rather an implementation issue. If NFS were implemented totally in userspace (like FTP), it would not hang the entire system when something breaks. On the other hand, it would be much slower than it is, therefore it would be unsuitable for a lot of use-cases where it is used now.

NFS hangs the system on failures because it is a shoddy implementation, not because it happens to be implemented in kernel space.

I think the old CVS quote by Tom Lord applies here:

  CVS has some strengths. It's a very stable piece of
  code, largely because nobody wants to work on it anymore.

Re: Magical Block Store: Why EBS Can't Work

#40

Post by CPlatypus »

jamesaguilar wrote: post #31
sigil wrote: post #30

Earlier quoted context omitted.

> Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect. You mean, in that case tolerance to partition and availability should be perfect. > Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are lower than the odds of you get…

> You mean, in that case tolerance to partition and availability should be perfect. No. If a network is never partitioned, you don't need to write algorithms that can tolerate partitions. Therefore consistency and availability are possible. > So this is the really interesting question. All the CAP theorem says is that (C,A,P) != (1.0,1.0,1.0). How close to (1.0,1.0,1.0) could we make (C,A,P)? If infinitely close, the…

Actually, C/A/P are more like variables you can multiply together. Even more accurately, they define a three-dimensional space with the CAP theorem (especially in Gilbert and Lynch's formal proof) only saying that you cannot be at the point that represents the maximum of all three. There are definitely tradeoffs or "mode switches" that can be made between C and A, and some even believe that you can give up some P to get more C/A. Personally I believe that the probability of partition is always non-zero in even the best designed and fully provisioned networks, even including those that are econonomically infeasible, and that those who choose to treat a small probability as zero to capture a performance advantage or position themselves as "more consistent" than competitors are being a bit dishonest, but those tradeoffs are being made.
Post reply on HN