Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Petabytes on a budget: How to build cheap cloud storage

blog.backblaze.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Petabytes on a budget: How to build cheap cloud storage

#81

Post by wmf »

lsc wrote: post #77
Devilboy wrote: post #75

Earlier quoted context omitted.

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive…

But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.

To get reliability without replication, it's not enough to use enterprise drives; you need redundant controllers that are multipathed to the disks. This is fairly expensive.

If you want a shared-nothing cluster with less than 3x overhead you can use erasure codes. The software complexity is significant, but at scale it should be the cheapest option. http://cleversafe.org/ or http://allmydata.org/trac/tahoe

Re: Petabytes on a budget: How to build cheap cloud storage

#82

Post by uhgygghhj »

Periodic wrote: post #53
jbellis wrote: post #35

Earlier quoted context omitted.

> While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.

This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of des…

I did this as an inhouse backup for a data warehousing app. Just slapping 4 ide cards into a case and putting 16x250gb IDE drives on them resulted in a system that would copy about 1 disk worth before hanging with some fault or suddenly dropping to 1% speed.

Just because you can in theory hook 40 drives to n cards doesn't mean it will work - well done to them

Re: Petabytes on a budget: How to build cheap cloud storage

#83

Post by uhgygghhj »

Devilboy wrote: post #75
lsc wrote: post #69

Earlier quoted context omitted.

as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows yo…

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive…

Does anyone use raid if they could use ZFS ? These people can be on BSD and so could use ZFS for free

Re: Petabytes on a budget: How to build cheap cloud storage

#84

Post by lsc »

wmf wrote: post #81
lsc wrote: post #77

Earlier quoted context omitted.

But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.

To get reliability without replication, it's not enough to use enterprise drives; you need redundant controllers that are multipathed to the disks. This is fairly expensive. If you want a shared-nothing cluster with less than 3x overhead you can use erasure codes. The software complexity is significant, but at scale it should be the cheapest option. http://cleversafe.org/ or http://allmydata.org/trac/tahoe

One solution I'm looking at is essentially mirrored raid5 over iscsi but it gets around your raid card single point of failure; the idea is that I have 2 zfs boxes (to begin/for testing these are OpenSolaris xen guests that control the spare drive slots I have in boxes that are primarily doing other things.) on each of those, i zraid the available drives. I then export iscsi luns of some standard size. The client box takes an iscsi lun from each opensolaris box and mirrors them with whatever software raid stack the client likes.

The big win here is caching... each of the opensolaris boxes can be using write-back caching, because all data is mirrored to the other opensolaris box. much like a NetAPP or EMC with dual heads, I don't have to worry about disk inconsistencies caused by write-back cache unless both drives fail. (of course, your 'real' dual-headed SAN will switch to write-through caching if one head dies, and doesn't require a full rebuild when the other head comes back online, but I can't afford a 'real' dual-headed SAN.)

Rebuild times, I imagine, will be quite significant.

Now, I am very leery of the performance of this system (being as these are xen instances, it wouldn't surprise me if it was unsuitable for anything but tape replacement) but I haven't tried yet. It seems, though that it would work just fine if I was not too cheap to use real hardware.

If you wanted to add complexity, you could have 3 or more of these opensolaris boxes, and use software raid5 on the client to save space.

Re: Petabytes on a budget: How to build cheap cloud storage

#85

Post by lsc »

Devilboy wrote: post #80
lsc wrote: post #77

Earlier quoted context omitted.

But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.

In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build…

raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well.

On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.)

The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)

Re: Petabytes on a budget: How to build cheap cloud storage

#87

Post by tzury »

PStamatiou wrote: post #39
jbellis wrote: post #35

Earlier quoted context omitted.

> While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.

I would have some smart-ass comment here but I just read your HN profile. Good day, Sir.

smart move ;-)

Re: Petabytes on a budget: How to build cheap cloud storage

#88

Post by Devilboy »

lsc wrote: post #85
Devilboy wrote: post #80

Earlier quoted context omitted.

In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build…

raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well. On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anythin…

Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.

Re: Petabytes on a budget: How to build cheap cloud storage

#89

Post by sho »

Periodic wrote: post #54
cperciva wrote: post #12

Earlier quoted context omitted.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

That's just the cost of hardware. How much more do you think it would cost if they included all their research and development, installation and assembly hours, etc? That is just the cost to buy all the components they have. If you budgeted that in you'd just have a bunch of boxes at your office that weren't even wired up. On the other hand, from Dell or Sun you at least have all the hardware in the chassis, if not s…

So you're currently paying $800k+ per box to install Solaris?

Tell you want, since you're a fellow HNer, I'll save you some money and do it for half that. I'll even throw in some basic configuration. You're welcome!

Re: Petabytes on a budget: How to build cheap cloud storage

#90

Post by Andys »

gstar wrote: post #9

They might have missed a trick here. To address vibration, acoustics and gyroscopic effect, what I've seen done in highly dense enclosures is to rotate every second drive around 180 degrees in a bit of a shotgun approach to balancing stuff. Still, awesome.

Unfortunately the SATA backplanes are only aligned one way, so they could only rotate a set of 5 drives at once.
Post reply on HN