Petabytes on a budget: How to build cheap cloud storage • Hacker News Board

Re: Petabytes on a budget: How to build cheap cloud storage

#101

Post by mmt » Wed, Sep 02, 2009, 11:58 PM UTC

lsc wrote: post #67

mmt wrote: post #60

Earlier quoted context omitted.

As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time. The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart. I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough d…

you might also want to check out coraid. it's at a similar price point. http://www.coraid.com/PRODUCTS/SR2421 - now, I've heard mixed experiences with it, and have no personal experience, but if you are talking about wiring up something yourself, it's definitely something to consider. Personally, I'm working on a system whereby I just fill up otherwise unused slots on my chassis and use that for backup. The system is…

Hmm.. looks suspiciously like exactly the SuperMicro case, for about twice the price of rolling ones own.

Re: Petabytes on a budget: How to build cheap cloud storage

#102

Post by lsc » Thu, Sep 03, 2009, 12:02 AM UTC

cakeface wrote: post #57

cperciva wrote: post #12

Earlier quoted context omitted.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

Another thing that they aren't taking into account over S3 is power costs, physical datacenter rent costs, and bandwith hookup. Still its probably a lot cheaper but important to note that they are comparing apples and apple cider.

what does s3 charge for transfer? $0.10 per gigabyte? I bet that covers much of the data center costs.

Re: Petabytes on a budget: How to build cheap cloud storage

#103

Post by mmt » Thu, Sep 03, 2009, 12:05 AM UTC

wmf wrote: post #68

mmt wrote: post #60

Earlier quoted context omitted.

As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time. The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart. I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough d…

Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card. I don't think a single SAS card can support more than 128 (or maybe 256 if you're lucky) disks. This doesn't affect your math much, but I wouldn't want someone to try this at home and be disappointed. If you're not willing to build a custom case and don't feel like suffering the vagaries of SA…

From experience, I can safely say that luck has nothing to do with it. LSI MegaSASes support more than 240, so it doesn't affect my math at all.

Similarly, from "at home" experience, I can also safely say that luck factors prominently in the vagaries of SATA port multipliers. The reason I even made the attempt was a lower price per port. This is why I'm so started that a solution based on them would end up being more expensive than a SAS-based one.

Of course, if the camel of performance gets its nose into the tent of ones requirements, it has to be the SAS way or the highway.

Re: Petabytes on a budget: How to build cheap cloud storage

#104

Post by mmt » Thu, Sep 03, 2009, 12:21 AM UTC

Periodic wrote: post #53

jbellis wrote: post #35

Earlier quoted context omitted.

> While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.

This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of des…

I don't think it takes quite as much effort as you imply, since a fairly basic storage array from one of the major vendors is $100k. (And then there's "maintenance" and software upgrades, but that's a separate matter).

Even if you got a $100/hr consultant, $100k would be half a year. I suggest it could be done in half that time, by someone who has suitable background. Someone with the specific expertise should be able to do it in 6 weeks elapsed and 2 weeks of full time work, tops. (bt; dt)

Re: Petabytes on a budget: How to build cheap cloud storage

#105

Post by lsc » Thu, Sep 03, 2009, 12:37 AM UTC

mmt wrote: post #101

lsc wrote: post #67

Earlier quoted context omitted.

you might also want to check out coraid. it's at a similar price point. http://www.coraid.com/PRODUCTS/SR2421 - now, I've heard mixed experiences with it, and have no personal experience, but if you are talking about wiring up something yourself, it's definitely something to consider. Personally, I'm working on a system whereby I just fill up otherwise unused slots on my chassis and use that for backup. The system is…

Hmm.. looks suspiciously like exactly the SuperMicro case, for about twice the price of rolling ones own.

from what I understand the coraid systems are plan9-based, and only have a gigabyte of cache ram.

If I only needed one I'd seriously consider it, as my time, for setting up one unit and working out the kinks, would unquestionably be more than that, and they have quite a bit of experience working out the problems by now, even if I made my system redundant, their system would unquestionably be more reliable the first iteration.

However, I plan on needing a bunch more, so it probably makes sense for me to roll my own, work out the kinks, and hopefully end up with something cheaper that has a lot more cache.

Re: Petabytes on a budget: How to build cheap cloud storage

#106

Post by jwilliams » Thu, Sep 03, 2009, 1:42 AM UTC

yellowbkpk wrote: post #98

jwilliams wrote: post #86

Awesome. Brings up a few questions for me (sure there are answers, just curious really). Why a Core 2? An Atom mobo would be lower power and cheaper. Why 4Gb? Seems like an overkill. They are using a HD to boot. Couldn't they boot of a USB key?

I don't know for sure, but there might not be an Atom-based board that has as many PCI/PCIe slots.

Good point. The one I've got only has one PCI in fact.