Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Petabytes on a budget: How to build cheap cloud storage

blog.backblaze.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Petabytes on a budget: How to build cheap cloud storage

#51

Post by lsc »

notaddicted wrote: post #15
tsuraan wrote: post #7

67TB of storage, with 4GB of cache. I'd really love to see some performance numbers versus the way-too-expensive competition. If the systems are being used as tape drive replacements, I could see this working well, but as an actual NAS-like device, I can't imaging how it could perform acceptably. Of course, if those Intel motherboards have the dual 1Gb/s NICs that Intel boards generally do, it will probably take a wh…

http://www.wolframalpha.com/input/?i=(67+TB)+%2F+(2*(1+Gb%2F... ) It'll take 3 days at the theoretical max of the networking equipment to read/write the 67TB. The overhead of HTTPS constrains the networking so this is too low. I'd expect that their internet connection (i.e. in/out of the data center) is the real bottleneck. I believe that the system is being used as a tape drive replacement.

If the load is relatively sequential, then yeah. But if the load is mostly random, then there is an argument for having more cache.

Re: Petabytes on a budget: How to build cheap cloud storage

#52

Post by blasdel »

spydez wrote: post #6
edw519 wrote: post #3

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc. They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.

No, they aren't going Google's route!

Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters.

Backblaze's system is going to have several orders of magnitude less bandwidth:

  way too many drives in one server (some of them on the PCI bus!)
  use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s)
  RAID6 with too many drives per array (15-way XOR is no fun)
  why use JFS?
  only access is via HTTPS, not clear if SSL is done on the pod

Re: Petabytes on a budget: How to build cheap cloud storage

#53

Post by Periodic »

jbellis wrote: post #35
PStamatiou wrote: post #8

Earlier quoted context omitted.

Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh That being said - did seagate ever fix their firmware issue on t…

> While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.

This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of design over many systems, but for smaller companies it just isn't feasible.

We hacker types love to think that we could do the same thing in no time with little budget, and I'm sure we could get a first approximation. But the devil is in the details. Debugging the complex interaction of 20 different hardware components is not my idea of fun.

Hats off to them, particularly for sharing.

Re: Petabytes on a budget: How to build cheap cloud storage

#54

Post by Periodic »

cperciva wrote: post #12
edw519 wrote: post #3

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

That's just the cost of hardware. How much more do you think it would cost if they included all their research and development, installation and assembly hours, etc?

That is just the cost to buy all the components they have. If you budgeted that in you'd just have a bunch of boxes at your office that weren't even wired up. On the other hand, from Dell or Sun you at least have all the hardware in the chassis, if not some basic configuration and OS install done for you. Go with Amazon and you've already got it racked up in a data center with dedicated techs and replacement parts on hand.

It isn't a fair comparison.

Re: Petabytes on a budget: How to build cheap cloud storage

#55

Post by Periodic »

swombat wrote: post #10

Wow. Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)

The cost they are quoting is the hardware cost. It isn't including developer time, housing costs, maintenance, etc.

Hardware is cheap, power and developers aren't.

Re: Petabytes on a budget: How to build cheap cloud storage

#56

Post by rantfoil »

Periodic wrote: post #53
jbellis wrote: post #35

Earlier quoted context omitted.

> While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.

This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of des…

Secondary revenue streams anyone? I know we'd be interested in becoming a customer if Backblaze at some point decided to monetize this part of their knowledge too.

This seems like an opportunity to disrupt from the low end of the market for some scrappy enterprising engineers. If they don't do it, someone should.

Re: Petabytes on a budget: How to build cheap cloud storage

#57

Post by cakeface »

cperciva wrote: post #12
edw519 wrote: post #3

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

Another thing that they aren't taking into account over S3 is power costs, physical datacenter rent costs, and bandwith hookup. Still its probably a lot cheaper but important to note that they are comparing apples and apple cider.

Re: Petabytes on a budget: How to build cheap cloud storage

#58

Post by antonovka »

jacquesm wrote: post #19
swombat wrote: post #10

Wow. Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)

Apparently they use HTTPS for input/output using tomcat and some custom application. That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ? quote from the article: "A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debia…

That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ?

To help prevent a network compromise from resulting in storage management compromise? Just because something is internal doesn't mean it's safe. Once a host/network segment is compromised, you don't want it to be easy to jump to the next.

Otherwise, you've built M&M security. Hard candy shell on the outside, soft gooey chocolate insides. Mmm.

Re: Petabytes on a budget: How to build cheap cloud storage

#59

Post by cheriot »

blasdel wrote: post #52
spydez wrote: post #6

Earlier quoted context omitted.

Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc. They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.

No, they aren't going Google's route! Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters. Backblaze's system is going to have several order…

I'm guessing that backups are accessed rather infrequently and that someone recovering their data won't mind a few seconds in lag time. This isn't my area of expertise, but their trade off seems sensible. It's just a different scenario than what google is dealing with.

Re: Petabytes on a budget: How to build cheap cloud storage

#60

Post by mmt »

edw519 wrote: post #3

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time.

The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart.

I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough demand. Obviously, this is starting to change.

I am, however, startled that a driveless pod is so expensive ($2467 or $54.82 per drive), considering that external performance is, effectively, limited by gigabit Ethernet.

Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card. Add another $1k for mobo and cables, and that's $11300 for 240 drives or $47.08 per drive, without having to do a custom case.

Granted, it takes up almost twice the rack space, but it's still a decent power density at 50W/U AC (assuming 6.5W per drive DC).

It takes 52 groups of 13 1.5TB disks (Hmm.. 13 pops up an awful lot in their setup) to make a PB using their pods. 51 groups means 17 pods is .9945TB for $134k ($134.5/PB).

Using my pods, it would be .936TB for $120.3k ($128.5/PB), but what's $6k/PB between friends?

Post reply on HN