Live data from Hacker News

Petabytes on a budget: How to build cheap cloud storage

blog.backblaze.com

21–30 of 106 posts

Re: Petabytes on a budget: How to build cheap cloud storage

#21
What happens when one of the PSU's fails catastrophically, pushing a big electrical surge through the hard disks, frying half of them?

I didn't see anything in here that discusses that eventuality? And when you have that many servers, it IS going to happen at some point...

Re: Petabytes on a budget: How to build cheap cloud storage

#22
post #10

Wow. Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)

It seems to me that their business model, and their requirements, and their implementation solve one problem, and they do it well. The problem backblaze is solving is backups. Their front-end software sends a pile of data to the datacenter, where it is written en-masse. Should restore ever be needed, the data is then transfered back en-masse. Both of these operations have needs that are different from an S3 type service. Backup/restore can have latencies built in, can have slow seek times, etc. This allows them to take advantage of the lower end hardware.

For example each of these "pods" only has 4GB of RAM. If I was doing lots of random I/O on 67TB of drives, I would want a heck of a lot more ram for caching efficiencies.

Re: Petabytes on a budget: How to build cheap cloud storage

#23
post #17

Earlier quoted context omitted.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

Interesting, but Amazon doesn't guarantee most of those things. They guarantee a 99.99% uptime, but that isn't counting a complete datacenter failure. In fact, it sounds to me as if they have a similar set up to the backblaze people. 99.99% uptime means roughly 1 hour per year of downtime. I don't know what the specific failure rates on the components are, but it seems reasonable that A) the data drives are hot-swapp…

>but that isn't counting a complete datacenter failure

I believe that S3 replicates to multiple physical locations. So, while you might experience downtime, you probably won't experience data loss.

Re: Petabytes on a budget: How to build cheap cloud storage

#24
post #10

Wow. Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)

Apparently they use HTTPS for input/output using tomcat and some custom application. That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ? quote from the article: "A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debia…

One thought I had regarding that -- their client talks to a "main server" management system, which talks to each pod on using the tomcat layer. The "main server" then replies to the client with an IP or similar to send data to directly. Since this goes over the web, https is used for security. Since https is already being used, it is not so difficult to use it all the way for authentication etc.

Re: Petabytes on a budget: How to build cheap cloud storage

#26

What happens when one of the PSU's fails catastrophically, pushing a big electrical surge through the hard disks, frying half of them? I didn't see anything in here that discusses that eventuality? And when you have that many servers, it IS going to happen at some point...

It is called a fuse.

Re: Petabytes on a budget: How to build cheap cloud storage

#27
post #17

Earlier quoted context omitted.

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids,…

Interesting, but Amazon doesn't guarantee most of those things. They guarantee a 99.99% uptime, but that isn't counting a complete datacenter failure. In fact, it sounds to me as if they have a similar set up to the backblaze people. 99.99% uptime means roughly 1 hour per year of downtime. I don't know what the specific failure rates on the components are, but it seems reasonable that A) the data drives are hot-swapp…

Re. geographical redundancy, see comment http://news.ycombinator.com/item?id=422574 :

"Amazon keeps at least 3 copies of your data (which is what you need for high reliability) in at least 2 different geographical locations. "

I can't find the original source supporting that statement, but I also know it to be true based on direct contact with AWS team. (I'm using AWS since private alpha of EC2 in 2005)

Re: Petabytes on a budget: How to build cheap cloud storage

#28

What happens when one of the PSU's fails catastrophically, pushing a big electrical surge through the hard disks, frying half of them? I didn't see anything in here that discusses that eventuality? And when you have that many servers, it IS going to happen at some point...

It's addressed in the section "A Backblaze Storage Pod is a Building Block"

From the article:

When you run a datacenter with thousands of hard drives, CPUs, motherboards, and power supplies, you are going to have hardware failures it’s irrefutable. Backblaze Storage Pods are building blocks upon which a larger system can be organized that doesn’t allow for a single point of failure.

Re: Petabytes on a budget: How to build cheap cloud storage

#29
What terrified me was the number of low-level issues they had to address. SATA protocol problems, custom-designed SATA cards? Wow. I'd have no idea how to begin with that stuff. As other posters have noted, this company's business is storage. How many PB must one host for such specialization to make sense?

Re: Petabytes on a budget: How to build cheap cloud storage

#30
What a pity, I'd like to have a way to store just a encrypted backup directly (via ssh) instead of installing a app which crawls my system. Oh did I mention I use Linux? sign

Does somebody know any other company which provides this solution? I don't trust other people to crawl my system and to "encrypt" it for me.

EDIT: Added Question

Post reply on HN