Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magical Block Store: Why EBS Can't Work

joyeur.com

Post reply on HN
Print view View original

51–56 of 56 posts

Re: Magical Block Store: Why EBS Can't Work

#51

Post by ssmoot »

CPlatypus wrote: post #48
ssmoot wrote: post #45

Earlier quoted context omitted.

ZFS is just the FS. But you know that already. The SAN solutions they migrated to are not ZFS based. Unless I'm mis-remembering (I read this a couple days ago) they were only using ZFS to slice LUNs. Point is, you're taking pot-shots at ZFS when the main thrust appears to be: "It was hard to make iSCSI reliable. Once we did, by buying expensive storage-vendor backed solutions, we found it wasn't financially compellin…

Perhaps you should RTFA. The author says explicitly that what they do now is "lean on ZFS" and "keep the network out of the storage solution" which made their provisioning more complex because they could no longer treat local disks as ephemeral (i.e. that data can't be assumed to exist anywhere else). I knew this when I wrote the GP. My whole point is that they treated it as a "networked storage is broken" problem ev…

They "lean on ZFS" for DAS.

Seriously. You tell me. What does that have to do with your rant on ZFS? It could have as well been an LSI controller doing RAID6. Or mdadm. Doesn't matter.

That's the evolved solution they came up with.

The "networked storage is broken" pitch actually comes in with the EMC/NetApp interim solution as well. I don't buy it either, but it's a joke to claim the problem was ZFS on the Zones when the Targets weren't running ZFS.

You're awfully prickly, but I didn't suggest it came down to "Enterprise" NAS vs DAS. I actually think networked storage is here to stay (and that's a good thing).

I have my doubts we'll see a stable, inexpensive (or free) Distributed or Clustered file-system ready to replace traditional solutions anytime soon. I'm happy to see people try though.

You clearly have an axe to grind with ZFS though. In my experience it's been by far more stable than any available Linux FS I've used. Pull the power again and again, replace and resilver all you want. Manage terabytes and don't worry about corruption. I wouldn't trust ext3/4fs for anything I couldn't stand to lose...

PS: http://en.wikipedia.org/wiki/Direct-attached_storage

"People who know storage". I don't see iSCSI on that list. Nor FCoE. DAS (at least according to Wikipedia) explicitly rules out switching. Which is how I've always viewed it.

Re: Magical Block Store: Why EBS Can't Work

#52

Post by lobster_johnson »

> It’s commonly believed that EBS is built on DRBD with a dose of S3-derived replication logic.

Actually, it was discovered some time ago (http://openfoo.org/blog/amazon_ec2_underlying_architecture.h...) that EBS probably used Red Hat's open-source GNDB: http://sourceware.org/cluster/gnbd/

Re: Magical Block Store: Why EBS Can't Work

#53

Post by CPlatypus »

ssmoot wrote: post #51
CPlatypus wrote: post #48

Earlier quoted context omitted.

Perhaps you should RTFA. The author says explicitly that what they do now is "lean on ZFS" and "keep the network out of the storage solution" which made their provisioning more complex because they could no longer treat local disks as ephemeral (i.e. that data can't be assumed to exist anywhere else). I knew this when I wrote the GP. My whole point is that they treated it as a "networked storage is broken" problem ev…

They "lean on ZFS" for DAS. Seriously. You tell me. What does that have to do with your rant on ZFS? It could have as well been an LSI controller doing RAID6. Or mdadm. Doesn't matter. That's the evolved solution they came up with. The "networked storage is broken" pitch actually comes in with the EMC/NetApp interim solution as well. I don't buy it either, but it's a joke to claim the problem was ZFS on the Zones whe…

You're really not getting it, are you? I never said ZFS was the problem, as you seem to think. I'm just saying it's not the solution either. It's a crappy solution, failing to protect against host failures and creating myriad problems in provisioning around the fact that each VM's storage is stranded on one node until it's explicitly copied somewhere else. And if you don't think there are decent distributed filesystems out there, you're just not keeping up with the field and shouldn't be commenting on it.

Re: Magical Block Store: Why EBS Can't Work

#54

Post by ssmoot »

CPlatypus wrote: post #53
ssmoot wrote: post #51

Earlier quoted context omitted.

They "lean on ZFS" for DAS. Seriously. You tell me. What does that have to do with your rant on ZFS? It could have as well been an LSI controller doing RAID6. Or mdadm. Doesn't matter. That's the evolved solution they came up with. The "networked storage is broken" pitch actually comes in with the EMC/NetApp interim solution as well. I don't buy it either, but it's a joke to claim the problem was ZFS on the Zones whe…

You're really not getting it, are you? I never said ZFS was the problem, as you seem to think. I'm just saying it's not the solution either. It's a crappy solution, failing to protect against host failures and creating myriad problems in provisioning around the fact that each VM's storage is stranded on one node until it's explicitly copied somewhere else. And if you don't think there are decent distributed filesyste…

I don't think I am getting it no. You don't think ZFS is the problem?

So you aren't calling ZFS a "crappy solution"? Just the DAS usage?

What is your gripe exactly then? The overblown critique of networked storage? Well we agree on that at least then. I think.

Honestly, with all the "read the fucking article", it's-not-DAS, oh-it-is, CloudFS is way moar better than ZFS, I never said ZFS sucked, "Bryan ZFS Cantrill is a jackass" you've left me absolutely bewildered at what your intended point (if any) actually is?

For the record, my only comment on (free) distributed filesystems (that aren't vendor-locked and actually unusable to me) is that I wouldn't personally trust them with my data. Not until they have the features I need, and then are running out in the wild, widley deployed for a couple years so I'm not a guinea pig.

I'll even throw you a bone: Even just last year ZFS was having major melt-downs when a new inadequately vetted feature was added. A few years ago it wasn't uncommon to face corruption when trying to do fairly routine things managing disks. Bugs can and do happen.

Maybe CloudFS, or Gluster is ready for prime-time, housing terabytes of data reliably and never making a misstep. I just don't think it's smart to bet your business on it. Not at least without a plan B since moving data around isn't an option when you're down and have terabytes you need to get back online.

Re: Magical Block Store: Why EBS Can't Work

#55

Post by mkramlich »

edw wrote: post #44
ssmoot wrote: post #42

Earlier quoted context omitted.

I don't know how NFS keeps coming up. It's an entirely different use case. It doesn't help the credibility of a critique on networked block storage to harp on a vendor specific implementation of a technology that doesn't even operate in the same sphere. An NFS server is very simple. With NFS on it's own VLAN, and some very basic QoS, there's no reason an NFS server should be the weak point in your infrastructure. Esp…

I brought up NFS because it's an example of a service that implements an abstraction but does so in a way that undermines the assumptions of the implemented abstraction. I do not disagree that local disks are an unrealistic strategy for creating a scalable, fault-tolerant system. The disk abstraction is of limited utility when creating such systems, because "disk thinking" leads to giving in to seductive assumptions…

> I brought up NFS because it's an example of a service that implements an abstraction but does so in a way that undermines the assumptions of the implemented abstraction.

yep. NFS and the like make you more vulnerable to the Fallacies of Distributed Computing.

Re: Magical Block Store: Why EBS Can't Work

#56

Post by jamesaguilar »

CPlatypus wrote: post #40
jamesaguilar wrote: post #31

Earlier quoted context omitted.

> You mean, in that case tolerance to partition and availability should be perfect. No. If a network is never partitioned, you don't need to write algorithms that can tolerate partitions. Therefore consistency and availability are possible. > So this is the really interesting question. All the CAP theorem says is that (C,A,P) != (1.0,1.0,1.0). How close to (1.0,1.0,1.0) could we make (C,A,P)? If infinitely close, the…

Actually, C/A/P are more like variables you can multiply together. Even more accurately, they define a three-dimensional space with the CAP theorem (especially in Gilbert and Lynch's formal proof) only saying that you cannot be at the point that represents the maximum of all three. There are definitely tradeoffs or "mode switches" that can be made between C and A, and some even believe that you can give up some P to…

> they define a three-dimensional space with the CAP theorem (especially in Gilbert and Lynch's formal proof) . . .

That's weird because I've read the proof and they speak only of boolean instances of C, A, and P in the proof. They give no examples of systems where any of the three variables have values other than zero or one.

Post reply on HN