Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magical Block Store: Why EBS Can't Work

joyeur.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Magical Block Store: Why EBS Can't Work

#21

Post by edw »

This awesome entry perfectly captures why I have always hated NFS. I can deal with the possibility that if a machine's hard drive dies, my system is going to have a very hard time continuing to operate in a normal manner, but then NFS comes along, and you realize that all sorts of I/O operations that previously employed a piece of equipment that failed once every two and a half years now depend on a working network with a working NFS server on that network, and the combination of that network and that server are orders of magnitude less reliable.

And now you have situations on a regular basis where you type "ls" and you shell hangs and not even "kill -9" is going to save you. And you go back to using FTP or some other abstraction that does not apply 40,000 hour MTBF thinking to equipment that disappears for coffee breaks daily.

Re: Magical Block Store: Why EBS Can't Work

#22

Post by chubot »

edw wrote: post #21

This awesome entry perfectly captures why I have always hated NFS. I can deal with the possibility that if a machine's hard drive dies, my system is going to have a very hard time continuing to operate in a normal manner, but then NFS comes along, and you realize that all sorts of I/O operations that previously employed a piece of equipment that failed once every two and a half years now depend on a working network w…

A thousand times yes. My first thought when hearing about the EBS outage was "wow that seems more fragile than NFS, no wonder it failed spectacularly." NFS presents you with this nice familiar file system interface, and then random sys admins and programmers start creating a tangled mess of dependencies by dropping stuff there, without regard to what happens when it fails. Like the EBS outage, the failures tend to surprise people.

The great quote by Leslie Lamport: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

And this was an excellent and honest article about faulty programming abstractions. It's basically bashing you over the head with the "Fallacies of distributed computing". Don't silently turn local operations into remote operations. They're not the same thing and have to be treated differently at all levels.

Even Werner Vogels wrote a diatribe against "transparency", which is the same issue by another name: http://scholar.google.com/scholar?cluster=700969849916494972...

So I wonder what he thinks of this architectural choice. You have to give up something when communicating over the network. Vogels seems to have chosen consistency rather than availability in his designs. This paper was a turning point in his research. Its candor surprised me.

The file system interface does not let you relax consistency, so by default you have chosen availability. As the Joyent guys honestly remarked, this often has to be learned the hard way.

Re: Magical Block Store: Why EBS Can't Work

#23

Post by sigil »

blantonl wrote: post #15
SoftwareMaven wrote: post #12

Many things in software are impossible magic, until they are not. His argument boils down to "it is a hard problem that nobody has solved yet." That doesn't mean nobody will ever solve it. Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it.

Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it. That presumption assumes that the application is being used as the right tool to resolve the problem. And it also assumes that "the problem" is a finite and solvable item.

> And it also assumes that "the problem" is a finite and solvable item.

Yes. To make this a bit more concrete, if "the problem" is making distributed storage look and behave exactly like local storage, the CAP Theorem has something to say about its solvability.

Re: Magical Block Store: Why EBS Can't Work

#24

Post by jinushaun »

spullara wrote: post #4

Also worth noting is that Amazon isn't forcing you to use EBS. They also have tons of fast local storage available to RAID as you wish.

Except Amazon does... Their free micro instance only supports EBS. For some of us who just want to dip our toes into AWS without too much money or commitment, EBS is our only choice.

Re: Magical Block Store: Why EBS Can't Work

#25

Post by jamesaguilar »

sigil wrote: post #23
blantonl wrote: post #15

Earlier quoted context omitted.

Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it. That presumption assumes that the application is being used as the right tool to resolve the problem. And it also assumes that "the problem" is a finite and solvable item.

> And it also assumes that "the problem" is a finite and solvable item. Yes. To make this a bit more concrete, if "the problem" is making distributed storage look and behave exactly like local storage, the CAP Theorem has something to say about its solvability.

Depends. Local storage is also not perfectly available. If the network is reliable, you can probably get availability high enough that the system feels "close enough" to how local storage feels. Today's networks aren't that reliable, but someday there may be enough redundancy and bandwidth for this to happen.

Re: Magical Block Store: Why EBS Can't Work

#27

Post by sciurus »

johnb wrote: post #19

It's funny how disk abstractions get you every time. We used to store and process all of our uploads from our rails app on a GFS partition. GFS behaved like a normal disk most of the time, but we started having trouble processing concurrent uploads and couldn't replicate in dev. It turned out so GFS could work at all, it had different locking than regular disks. Every time you created a new file it had to lock the co…

FWIW, this behavior is explained early on in the documentation for GFS2.

Re: Magical Block Store: Why EBS Can't Work

#28

Post by sigil »

jamesaguilar wrote: post #25
sigil wrote: post #23

Earlier quoted context omitted.

> And it also assumes that "the problem" is a finite and solvable item. Yes. To make this a bit more concrete, if "the problem" is making distributed storage look and behave exactly like local storage, the CAP Theorem has something to say about its solvability.

Depends. Local storage is also not perfectly available. If the network is reliable, you can probably get availability high enough that the system feels "close enough" to how local storage feels. Today's networks aren't that reliable, but someday there may be enough redundancy and bandwidth for this to happen.

> Local storage is also not perfectly available.

Technically true, although you don't have to contend with the consistency or partitioning factors in the local disk case -- there's only one copy of the state. This means you can focus on making the availability factor as close to 1.0 as possible.

This may not be the case when you're forced to balance all three CAP factors. I sometimes wonder if a follow on result to CAP will be a "practical" (physical or information theoretic) limit like C x A x P = h. This is of course wildly unsubstantiated pessimism.

Also, I would gladly entertain any argument demolishing the "local disks are not subject to CAP" claim I made above by talking about read / write caches as separate copies of the local disk state.

Re: Magical Block Store: Why EBS Can't Work

#29

Post by jamesaguilar »

sigil wrote: post #28
jamesaguilar wrote: post #25

Earlier quoted context omitted.

Depends. Local storage is also not perfectly available. If the network is reliable, you can probably get availability high enough that the system feels "close enough" to how local storage feels. Today's networks aren't that reliable, but someday there may be enough redundancy and bandwidth for this to happen.

> Local storage is also not perfectly available. Technically true, although you don't have to contend with the consistency or partitioning factors in the local disk case -- there's only one copy of the state. This means you can focus on making the availability factor as close to 1.0 as possible. This may not be the case when you're forced to balance all three CAP factors. I sometimes wonder if a follow on result to C…

I doubt it. Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect. Although networks will never be perfectly reliable, nor machines, they seem to be getting more reliable. Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are lower than the odds of you getting struck by lightning, at which point you will have for practical purposes defeated the constraints of the CAP theorem.

Re: Magical Block Store: Why EBS Can't Work

#30

Post by sigil »

jamesaguilar wrote: post #29
sigil wrote: post #28

Earlier quoted context omitted.

> Local storage is also not perfectly available. Technically true, although you don't have to contend with the consistency or partitioning factors in the local disk case -- there's only one copy of the state. This means you can focus on making the availability factor as close to 1.0 as possible. This may not be the case when you're forced to balance all three CAP factors. I sometimes wonder if a follow on result to C…

I doubt it. Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect. Although networks will never be perfectly reliable, nor machines, they seem to be getting more reliable. Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are…

> Suppose that there is a network that is never partitioned, and machines connected to that network that never fail. In that case consistency and availability should be perfect.

You mean, in that case tolerance to partition and availability should be perfect.

> Perhaps someday we may be able to say that the odds of enough partitions or machine failures to make the system unavailable are lower than the odds of you getting struck by lightning, at which point you will have for practical purposes defeated the constraints of the CAP theorem.

So this is the really interesting question. All the CAP theorem says is that (C,A,P) != (1.0,1.0,1.0). How close to (1.0,1.0,1.0) could we make (C,A,P)? If infinitely close, then we have achieved perfection by the limit, and the CAP theorem is rather pointless. If not, then what is the numeric limit?

As you speculate, maybe the numeric limit on C x A x P is so close to 1.0 that the odds of seeing a consistency, availability, or partitioning problem are much smaller than getting hit by lightning.

Then again, maybe not. Who knows? ;)

To avoid sounding like a total crackpot, here is an interesting paper that explores the physical limits of computation:

http://arxiv.org/pdf/quant-ph/9908043v3

Post reply on HN