Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magical Block Store: Why EBS Can't Work

joyeur.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Magical Block Store: Why EBS Can't Work

#11

Post by spudlyo »

Trying to use a tool like iostat against a shared, network provided block device to figure out what your level of service your database is getting from the filesystem below it is an exercise in frustration that will get you nowhere.

This may be true under Solaris. Since 2.5 Linux has had /proc/diskstats and an iostat that shows the average i/o request latency (await) for a disk, network or otherwise. For EBS it's 40ms or less on a good day. On a bad day it's 500ms or more if your i/o requests get serviced at all.

Re: Magical Block Store: Why EBS Can't Work

#12

Post by SoftwareMaven »

Many things in software are impossible magic, until they are not. His argument boils down to "it is a hard problem that nobody has solved yet." That doesn't mean nobody will ever solve it.

Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it.

Re: Magical Block Store: Why EBS Can't Work

#13

Post by blantonl »

spullara wrote: post #4

Also worth noting is that Amazon isn't forcing you to use EBS. They also have tons of fast local storage available to RAID as you wish.

I agree. We run all of our MySQL and Mongo slave servers with local RAID-0 ephemeral storage. One dies? So what, we remove it from the pool and provision another.

Only our master and our slave backup server runs on EBS. We aren't as write oriented so we can live with some of the limitations of EBS, but we've even considered moving our master MySQL and Mongo servers to ephemeral storage and just relying on our slave back database server to run on EBS (for which we take freeze/snapshots of often). That server rarely ever falls more behind in relay updates.

Re: Magical Block Store: Why EBS Can't Work

#14

Post by count »

blantonl wrote: post #10
smhinsey wrote: post #6

Earlier quoted context omitted.

Do you mean RAIDing local instance storage? I haven't heard of this approach so far, do you have any links?

Depending on the instance that you have provisioned, you have anywhere from 1 to 4 ephemeral (local) disks available to your instance. They are typically available as /dev/sd[bcde] In centOS, implementing a RAID-0 block device across 2 ephemeral disks that is present on an m1.large instance can be done via the following: mdadm --create /dev/md0 --metadata=1.1 --level=0 --quiet --run -c 256 -n 2 /dev/sdb /dev/sdc You'…

Just remember the super important keyword:

ephemeral

WHEN your EC2 node disappears (and it will), you will lose everything on that RAID.

That's not a bad thing if you know it'll happen and plan for it, but do be aware of it.

Re: Magical Block Store: Why EBS Can't Work

#15

Post by blantonl »

SoftwareMaven wrote: post #12

Many things in software are impossible magic, until they are not. His argument boils down to "it is a hard problem that nobody has solved yet." That doesn't mean nobody will ever solve it. Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it.

Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it.

That presumption assumes that the application is being used as the right tool to resolve the problem. And it also assumes that "the problem" is a finite and solvable item.

Re: Magical Block Store: Why EBS Can't Work

#16

Post by cagenut »

Its really fascinating to watch amazon re-learn/re-implement the lessons IBM baked into mainframes decades ago. Once you get out of shared-nothing/web-scripting land you realize that I/O is much more important and difficult than cpu. What amazon calls EBS IBM has been calling "DASD" forever. I wonder if there are any crossover lessons that they haven't taken advantage of because there just aren't any old ibm'ers working at amazon.

Re: Magical Block Store: Why EBS Can't Work

#17

Post by blantonl »

cagenut wrote: post #16

Its really fascinating to watch amazon re-learn/re-implement the lessons IBM baked into mainframes decades ago. Once you get out of shared-nothing/web-scripting land you realize that I/O is much more important and difficult than cpu. What amazon calls EBS IBM has been calling "DASD" forever. I wonder if there are any crossover lessons that they haven't taken advantage of because there just aren't any old ibm'ers work…

IBM's implementation of DASD on the mainframe was always implemented under the assumption that it was a secondary storage medium for data. Meaning, it wasn't accessed often, and it wasn't implemented for top performance.

Think of a bridge between high performance disk and tape.

Re: Magical Block Store: Why EBS Can't Work

#19

Post by johnb »

It's funny how disk abstractions get you every time.

We used to store and process all of our uploads from our rails app on a GFS partition. GFS behaved like a normal disk most of the time, but we started having trouble processing concurrent uploads and couldn't replicate in dev.

It turned out so GFS could work at all, it had different locking than regular disks. Every time you created a new file it had to lock the containing folder. We solved it by splitting our upload folder in 1000 sequential buckets and wrote each upload to the next folder along... but it took us a long time to stop assuming it was a regular disk.

Re: Magical Block Store: Why EBS Can't Work

#20

Post by epi0Bauqu »

blantonl wrote: post #13
spullara wrote: post #4

Also worth noting is that Amazon isn't forcing you to use EBS. They also have tons of fast local storage available to RAID as you wish.

I agree. We run all of our MySQL and Mongo slave servers with local RAID-0 ephemeral storage. One dies? So what, we remove it from the pool and provision another. Only our master and our slave backup server runs on EBS. We aren't as write oriented so we can live with some of the limitations of EBS, but we've even considered moving our master MySQL and Mongo servers to ephemeral storage and just relying on our slave b…

Do you run xlarge to get the extra ephemeral disks?
Post reply on HN