Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Petabytes on a budget: How to build cheap cloud storage

blog.backblaze.com

Post reply on HN
Print view View original

71–80 of 106 posts

Re: Petabytes on a budget: How to build cheap cloud storage

#71

Post by pmorici »

cakeface wrote: post #57
cperciva wrote: post #12

Earlier quoted context omitted.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

Another thing that they aren't taking into account over S3 is power costs, physical datacenter rent costs, and bandwith hookup. Still its probably a lot cheaper but important to note that they are comparing apples and apple cider.

On their graph it says they subtracted data center costs from the price of S3.

Re: Petabytes on a budget: How to build cheap cloud storage

#72

Post by pmorici »

tsuraan wrote: post #7

67TB of storage, with 4GB of cache. I'd really love to see some performance numbers versus the way-too-expensive competition. If the systems are being used as tape drive replacements, I could see this working well, but as an actual NAS-like device, I can't imaging how it could perform acceptably. Of course, if those Intel motherboards have the dual 1Gb/s NICs that Intel boards generally do, it will probably take a wh…

What if they used AndrewFS with SSD drives for cache on front end nodes.

Re: Petabytes on a budget: How to build cheap cloud storage

#74

Post by hyuen »

sh1mmer wrote: post #49

One of the smart guys in our (Y!) cloud team just point out something to me that hadn't occurred to me. This system is definitely optimized for backup. That totally make sense for Backblaze. However it's important to not compare this like for like with something like S3 which is optimized for much better read/write performance. At the basic level the cooling on this system seems minimal. Those tightly packed drives w…

I was thinking about the IOPS that you can sustain with this type of device, and as mentioned here, this is intended for backup, which may or may not be very good acting as a filer.

Probably you can get some high IOPS with all these drives in parallel, but probably not as much as the other more expensive ones (EMC, Netapp).

But I have to agree this is an awesome setup optimized for a specific application. Way to go

Re: Petabytes on a budget: How to build cheap cloud storage

#75

Post by Devilboy »

lsc wrote: post #69
PStamatiou wrote: post #8

Earlier quoted context omitted.

Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh That being said - did seagate ever fix their firmware issue on t…

as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows yo…

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique.

The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive since the drives make up most of the hardware cost.

Re: Petabytes on a budget: How to build cheap cloud storage

#76

Post by rarrrrrr »

papercrane wrote: post #31
sschueller wrote: post #25

How do you replace a drive? With so many drives it seems like a lot of effort to pull the entire server out to just access one drive.

They have three raid-6 volumes per machine, so they probably leave it in the rack until two of the volumes are unusable and then refurbish the machine.

I doubt it very much. When drives start failing out of a RAID6 array, you lose it's ability to automatically detect and correct single bit errors on disk.

With that much data, single bit errors will happen predictably.

If their usage patterns are anything like ours, disk failures are well under normal rates. Most backup data is stored and ignored. The drives just don't get much stress. Downing a machine for maintenance is probably acceptable to their usage pattern.

Re: Petabytes on a budget: How to build cheap cloud storage

#77

Post by lsc »

Devilboy wrote: post #75
lsc wrote: post #69

Earlier quoted context omitted.

as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows yo…

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive…

But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10.

Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.

Re: Petabytes on a budget: How to build cheap cloud storage

#78

Post by ciupicri »

gstar wrote: post #9

They might have missed a trick here. To address vibration, acoustics and gyroscopic effect, what I've seen done in highly dense enclosures is to rotate every second drive around 180 degrees in a bit of a shotgun approach to balancing stuff. Still, awesome.

This has sounded ok for me too, but when I've thought a bit about it I've realized that the drives must be synchronized so that each one would "cancel" each others vibrations. The vibrations generated by the drives might have indeed the same amplitude and (spatial and temporal) frequency, but I doubt that they'll have the same phase offset. For this to happen they would need to be started at _exactly_ the same time which I don't think that happens in practice.

Do you have some pictures of those enclosures?

Re: Petabytes on a budget: How to build cheap cloud storage

#80

Post by Devilboy »

lsc wrote: post #77
Devilboy wrote: post #75

Earlier quoted context omitted.

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive…

But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.

In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time.

With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish.

The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?

Post reply on HN