Petabytes on a budget: How to build cheap cloud storage • Hacker News Board

Re: Petabytes on a budget: How to build cheap cloud storage

#91

Post by Andys » Wed, Sep 02, 2009, 7:28 AM UTC

tricky wrote: post #11

effing love this. I wonder what the reliability stats on this setup is, though. Is it really cheaper to jam all those drives in one unit without redundant PSU's, MB's, or a boot drive? I'd guess you'd have to build at least 2 of these units and mirror them to get any sort of reliability. And, at that point, how long does it take to copy 58 TB across https? data is hard.

In sysadmin terms, they have aimed for data integrity over availability.

If a system goes down and they have to replace a disk or a power supply or motherboard, the data is still safe, and if by chance there's any $5/month users who need to restore data held on that one unit, well they can wait a few hours :-)

Re: Petabytes on a budget: How to build cheap cloud storage

#92

Post by Andys » Wed, Sep 02, 2009, 7:35 AM UTC

sh1mmer wrote: post #49

One of the smart guys in our (Y!) cloud team just point out something to me that hadn't occurred to me. This system is definitely optimized for backup. That totally make sense for Backblaze. However it's important to not compare this like for like with something like S3 which is optimized for much better read/write performance. At the basic level the cooling on this system seems minimal. Those tightly packed drives w…

Fast 120mm fans can move a pretty decent amount of air - up to 120 ft^3/min each, and they used three in parallel.

It would take about a dozen 80mm normal-speed computer fans to reach this.

Re: Petabytes on a budget: How to build cheap cloud storage

#93

Post by silverscreen » Wed, Sep 02, 2009, 7:59 AM UTC

Great read! Why a seperated boot disk? Why can't that be on the storage array?

Re: Petabytes on a budget: How to build cheap cloud storage

#94

Post by gstar » Wed, Sep 02, 2009, 8:13 AM UTC

lsc wrote: post #84

wmf wrote: post #81

Earlier quoted context omitted.

To get reliability without replication, it's not enough to use enterprise drives; you need redundant controllers that are multipathed to the disks. This is fairly expensive. If you want a shared-nothing cluster with less than 3x overhead you can use erasure codes. The software complexity is significant, but at scale it should be the cheapest option. http://cleversafe.org/ or http://allmydata.org/trac/tahoe

One solution I'm looking at is essentially mirrored raid5 over iscsi but it gets around your raid card single point of failure; the idea is that I have 2 zfs boxes (to begin/for testing these are OpenSolaris xen guests that control the spare drive slots I have in boxes that are primarily doing other things.) on each of those, i zraid the available drives. I then export iscsi luns of some standard size. The client box…

Are your target boxes virtual machines?

I always prefer the strategy where mirroring the data happens before the disk is presented to the VM. Have a look at GlusterFS and see if that might be an option? I know it wasn't playing nice with zfs for a while there, though.

Re: Petabytes on a budget: How to build cheap cloud storage

#95

Post by gstar » Wed, Sep 02, 2009, 8:22 AM UTC

ciupicri wrote: post #78

gstar wrote: post #9

They might have missed a trick here. To address vibration, acoustics and gyroscopic effect, what I've seen done in highly dense enclosures is to rotate every second drive around 180 degrees in a bit of a shotgun approach to balancing stuff. Still, awesome.

This has sounded ok for me too, but when I've thought a bit about it I've realized that the drives must be synchronized so that each one would "cancel" each others vibrations. The vibrations generated by the drives might have indeed the same amplitude and (spatial and temporal) frequency, but I doubt that they'll have the same phase offset. For this to happen they would need to be started at _exactly_ the same time w…

I don't have pictures. I can describe, though.

Two disks were mounted in a frame linearly, both screwed to the frame, with the power/sata connectors toward the middle of the frame, and one drive upside-down.

These cassettes were removable as a unit for hot-swap, and were inserted linearly into a half-deep 19" rack enclosure.

That the two drives were physically connected to the same frame, and removed and replaced as a unit would make it seem as if they would be started in phase. Now I'm not physicist, but I'm not 100% sure that's so important - if you have two gyroscopes contra-rotating and firmly connected, running at the same speed, surely they resist movement by sheer gyroscopic effect?

Re: Petabytes on a budget: How to build cheap cloud storage

#96

Post by ggruschow » Wed, Sep 02, 2009, 10:12 AM UTC

pmorici wrote: post #73

Seems along the same lines as Capricorn Tech. http://www.capricorn-tech.com/products.php

Not really. BackBlaze's post is about how to get on-line storage up for the lowest cost per bit. They appear to have done a great job at quadrupling the density of standard 1U setups for a similar price.

Capricorn appears to be in the business of selling standard 4 drive per 1U setups and support. The fact you have to contact their sales department to even get close to a price seems to indicate they're not competing on price.

Re: Petabytes on a budget: How to build cheap cloud storage

#97

Post by eleitl » Wed, Sep 02, 2009, 11:38 AM UTC

This made the Beowulf list yesterday, and below is what I wrote in response:

"Seagate ST31500341AS 1.5TB Barracuda 7200.11 SATA 3Gb/s 3.5″ Aargh! Should be definitely substituted by 2 TByte WD RE4 drive.

Today I've built a 32 TByte raw storage Supermicro box with X8DDAi (dual-socket Nehalem, 24 GByte RAM, IPMI), two LSI SAS3081E-R and OpenSolaris sees all (WD2002FYPS) drives so far (the board refuses to boot from DVD when more than 12 drives are in though probably to some BIOS brain damage, so you have to manually build a raidz-2 with all 16 drives in it once Solaris has booted up). The drives are about 3170 EUR sans VAT total for all 16, the box itself around 3000 EUR sans VAT. I presume Linux with RAID 6 would work (haven't checked yet), too, and if you need more you can use a cluster FS.

Maybe not as cheap as a Backblaze, but off-shelf (BTO) and you get what you pay for.

Re: Petabytes on a budget: How to build cheap cloud storage

#98

Post by yellowbkpk » Wed, Sep 02, 2009, 1:52 PM UTC

jwilliams wrote: post #86

Awesome. Brings up a few questions for me (sure there are answers, just curious really). Why a Core 2? An Atom mobo would be lower power and cheaper. Why 4Gb? Seems like an overkill. They are using a HD to boot. Couldn't they boot of a USB key?

I don't know for sure, but there might not be an Atom-based board that has as many PCI/PCIe slots.

Re: Petabytes on a budget: How to build cheap cloud storage

#99

Post by lsc » Wed, Sep 02, 2009, 5:23 PM UTC

Devilboy wrote: post #88

lsc wrote: post #85

Earlier quoted context omitted.

raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well. On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anythin…

Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.

Yeah. that's my thought. I'm at a couple racks now, but I want a solution that will scale beyond that. Thing is, to be honest, the coraid crap will probably be more reliable (and cheaper if you include my time) than stuff I build myself for the first few units, until I learn what I'm doing, but long-term, I could be significantly better off if my idea works as planned, as my units are potentially much cheaper and have a whole lot more cache (and cache redundancy) than the coraid.

On the other hand, if I handle the 'beta' poorly, like taking it out of beta early, there won't be a long term to worry about.

Re: Petabytes on a budget: How to build cheap cloud storage

#100

Post by lsc » Wed, Sep 02, 2009, 5:29 PM UTC

gstar wrote: post #94

lsc wrote: post #84

Earlier quoted context omitted.

One solution I'm looking at is essentially mirrored raid5 over iscsi but it gets around your raid card single point of failure; the idea is that I have 2 zfs boxes (to begin/for testing these are OpenSolaris xen guests that control the spare drive slots I have in boxes that are primarily doing other things.) on each of those, i zraid the available drives. I then export iscsi luns of some standard size. The client box…

Are your target boxes virtual machines? I always prefer the strategy where mirroring the data happens before the disk is presented to the VM. Have a look at GlusterFS and see if that might be an option? I know it wasn't playing nice with zfs for a while there, though.

Virtual private servers, yes. I use Xen.

And I agree, giving the user 2 block devices and expecting them to mirror is a very bad idea. I know this from experience. in an earlier setup, I'd give each user two block devices, one from each disk in the box. the idea being the user would mirror or not as appropriate. well, you loose a customers data, you loose the customer. Lesson learned.

But this is where my virtual setup gives me an edge. On every Xen server, I have a control guest or driver domain, the dom0. all disk and net I/O goes through the dom0 anyhow, so there wouldn't be much more overhead to me doing the mirroring on the Dom0, and passing the md through to the DomU (what the user controls.) I'm not adding another point of failure, either; if the dom0 chokes, all guests on the system will crash regardless.