Live data from Hacker News

Petabytes on a budget: How to build cheap cloud storage

blog.backblaze.com

41–50 of 106 posts

Re: Petabytes on a budget: How to build cheap cloud storage

#41
I have been searching high and low for something like this for months -- I have a number of clients that have been begging me for this, as well as something for my own needs.

I'm excitedly posting a link to this on my personal site, and today I have a lot of phone calls to make to clients.

Why is it that the best products and services are also the hardest to find when you're looking for them?

Re: Petabytes on a budget: How to build cheap cloud storage

#42
The price comparison with other solutions isn't really fair. The cost per PB of these storage building blocks is directly compared to complete storage solutions from companies like NetApp and EMC. In the end the cost of the complete solution they assemble these building blocks into may well be cheaper than solutions from other companies and that's the number they should be using for comparison.

Re: Petabytes on a budget: How to build cheap cloud storage

#43

Earlier quoted context omitted.

It's addressed in the section "A Backblaze Storage Pod is a Building Block" From the article: When you run a datacenter with thousands of hard drives, CPUs, motherboards, and power supplies, you are going to have hardware failures it’s irrefutable. Backblaze Storage Pods are building blocks upon which a larger system can be organized that doesn’t allow for a single point of failure.

Sounds to me like they haven't implemented this yet. I wouldn't want my data on that sort of solution. Backed up data would surely need a level of geographic redundancy.

In the next section they say they have implemented the machine level redundancy. As for geographic redundancy I think the idea is that you use them for that. As in you backup your data locally and then send it to them as a redundant copy.

Re: Petabytes on a budget: How to build cheap cloud storage

#45
post #3

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

This is not apples vs apples. Some of those items are living cows, and some are cooked steaks served with a side of mashed potatoes. I'm sure the cow looks cheap (per pound) until you are tasked with turning it into a steak.

Re: Petabytes on a budget: How to build cheap cloud storage

#46
I'd love to hear what they are doing to monitor their system. They didn't mention that. 3 out of 15 drives failures to loose a volume seems quite likely at that scale.

I'd like to know what levels of warnings and alarms they use with which system, e.g. nagios, etc.

Re: Petabytes on a budget: How to build cheap cloud storage

#47
I wonder about the single point of failure posed by the power supplies. One failed box is not a big deal (since I assume the data is replicated over several). But, what if they get a bad batch of supplies and see a relatively high failure rate? I wonder how high a power supply failure rate they can handle.

The need to stagger the power on of the two supplies poses a problem. What if power to a data center is lost? When power is restored, all box will try to start, blowing all fuses. Granted, this is a catastrophic event, so its frequency should be very low. But, this also seems like an area that could be automated.

Re: Petabytes on a budget: How to build cheap cloud storage

#48
post #47

I wonder about the single point of failure posed by the power supplies. One failed box is not a big deal (since I assume the data is replicated over several). But, what if they get a bad batch of supplies and see a relatively high failure rate? I wonder how high a power supply failure rate they can handle. The need to stagger the power on of the two supplies poses a problem. What if power to a data center is lost? Wh…

You should only use 75% of the rated capacity of a circuit, which means you have enough power to turn them all on at once.

some of the more expensive managed power supplies also support a staggered power on after power fail. But I don't worry about it; only using 75% of the power circuit solves that problem for me.

Re: Petabytes on a budget: How to build cheap cloud storage

#49
One of the smart guys in our (Y!) cloud team just point out something to me that hadn't occurred to me.

This system is definitely optimized for backup. That totally make sense for Backblaze. However it's important to not compare this like for like with something like S3 which is optimized for much better read/write performance.

At the basic level the cooling on this system seems minimal. Those tightly packed drives would sure get hot if they were all spinning a lot. More than that since they are using commodity consumer hardware, and they already used up their PCIe slots for the SATA controllers there isn't any place to add anything more than the gitabit (I assume) ethernet jack on the mobo. That means there throughput is limited.

Again, this is a great system for backup. Most of the data will just sit happily sipping little power. However, if you are thinking of this as equivalent to a filer, that's an unfair comparison.

Re: Petabytes on a budget: How to build cheap cloud storage

#50
post #7

67TB of storage, with 4GB of cache. I'd really love to see some performance numbers versus the way-too-expensive competition. If the systems are being used as tape drive replacements, I could see this working well, but as an actual NAS-like device, I can't imaging how it could perform acceptably. Of course, if those Intel motherboards have the dual 1Gb/s NICs that Intel boards generally do, it will probably take a wh…

Actually, that's the problem with most NetAPP/EMC foo, too. they are way light on the cache.

These guys only use 4GiB ram because they are saving money on the motherboard, I bet. Personally, if I were building it, I'd increase the cost by another grand or so and use dual low power opterons with 32GiB ram. (of course, that would also increase the space taken by the motherboard, so that would require some case redesigns. Still, opterons and registered ecc ddr2 are both incredibly cheap right now.)

Post reply on HN