Live data from Hacker News

Why we built a 40TB photo server in-house instead of using S3

tech.oyster.com

61–70 of 78 posts

Re: Why we built a 40TB photo server in-house instead of using S3

#61
post #55

A bit off-topic, but I wonder if anyone can comment on using software RAID rather than hardware RAID? I'd like to try it. I've been bitten by buggy hardware RAID controllers far too often (even high-dollar name brand gear). I know that all of the free *nix systems offer software RAID, I'm just curious how they perform.

This may be useful too, BackBlaze uses mdadm RAID for its storage:

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-h...

and version 2

http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...

Re: Why we built a 40TB photo server in-house instead of using S3

#62
post #57

Earlier quoted context omitted.

I can't find any documented evidence of data loss on S3. Can you?

S3 has lost data 3 times in its history.

Documented evidence. eg not http://www.quora.com/Has-Amazon-S3-ever-lost-data-permanentl... "All of this is my recollection of what I've been told friends, so it may not be accurate at all."

Re: Why we built a 40TB photo server in-house instead of using S3

#63
the author updated with this comment:

http://tech.oyster.com/how-to-build-a-40tb-file-server/?#com...

"Didn’t mean to give the impression that this is the only backup, it is not. It is the “warmest” one — first line. It does not share the same physical location with the primary storage box either, but is less than 2 miles away. So we can easily have a 2 hour recovery time without throwing away those astronomical monthly service fees. (Although many technologists will always prefer paying big bucks for the comfort of being cushioned from every angle by SLAs and such — nothing wrong with that, just a different approach.)

As far as TCO goes, it cannot get any lower since we already have one or two system guys handling all servers, as well as office workstations, etc.. This backup box takes up such a small fraction of their time that its almost negligible — several thousand annually at most. Same goes for power, etc — it is just one of many servers.

The disks are all Enterprise Class 2TB SATA-II, several different models. We were purchasing them right after the monsoon floods in Thailand constricted supply so our choices were somewhat limited as time was a factor.

Raid6 has come a long way since it’s early inception days, but is still a trade-off between raw storage capacity and processor utilization. HW RAID industry is now old enough to not have to wait for new products to mature as we used to when the technology itself was in its infancy. Old habits certainly die hard, but getting the “latest and greatest” was a conscious choice made for this specific problem, not submission to some immature fascination with “elite” new products, or however that may be.. This card has the best specs for Raid6 currently on the market — bottom line, period.

Big Kudos to all who made suggestions and participate in the discussion, keep it coming!"

Re: Why we built a 40TB photo server in-house instead of using S3

#64
post #22

Earlier quoted context omitted.

Better than going bankrupt.

You mean, better than going bankrupt until they go bankrupt when the single BOX goes down, right?

That single box is made of replaceable cheap parts. Only a fire can take it completely out.

Re: Why we built a 40TB photo server in-house instead of using S3

#65

Earlier quoted context omitted.

I've been doing basic RAID stuff for a while, most notably for a client that does physics modeling (and other things). So far, my favorite build is: - A box from abmx.com. Really nice people, good hardware. - An Areca RAID controller (might have to talk to them about this). - FreeBSD, which has good support for Areca controllers, thanks to Areca's friendliness towards the open source community. - ZFS. This particular…

Can you, or anyone else, give opinion about LSi 92xx MegaRAID cards vs ATTO r680 vs Areca? They all seem the same to me.

AFAIK IBM, HP, and Dell use nothing but LSI controllers in their servers; that may be a hint.

Re: Why we built a 40TB photo server in-house instead of using S3

#66
post #65

Earlier quoted context omitted.

Can you, or anyone else, give opinion about LSi 92xx MegaRAID cards vs ATTO r680 vs Areca? They all seem the same to me.

AFAIK IBM, HP, and Dell use nothing but LSI controllers in their servers; that may be a hint.

That's a good thing to know! Thanks.

Re: Why we built a 40TB photo server in-house instead of using S3

#67

Earlier quoted context omitted.

I've been doing basic RAID stuff for a while, most notably for a client that does physics modeling (and other things). So far, my favorite build is: - A box from abmx.com. Really nice people, good hardware. - An Areca RAID controller (might have to talk to them about this). - FreeBSD, which has good support for Areca controllers, thanks to Areca's friendliness towards the open source community. - ZFS. This particular…

Can you, or anyone else, give opinion about LSi 92xx MegaRAID cards vs ATTO r680 vs Areca? They all seem the same to me.

Unfortunately, no. I haven't used the other two.

Re: Why we built a 40TB photo server in-house instead of using S3

#68
post #64

Earlier quoted context omitted.

You mean, better than going bankrupt until they go bankrupt when the single BOX goes down, right?

That single box is made of replaceable cheap parts. Only a fire can take it completely out.

My company's data center has suffered a fire at least once in the past 5 years.

Re: Why we built a 40TB photo server in-house instead of using S3

#69
post #6

S3 gives you multi-host, multi-region redundancy. Putting it all in one box is asking for trouble. What if the raid controller grows a bug and corrupts on write? It's happened. What if there's a fire in the building that has both your server and your back-up? Eggs, meet basket!

S3 fails too, right? Didn't they have an internal network issue this year, and go down for hours? Anything you haven't tried, doesn't work. That's a truism in computing. I don't think Amazon tries failing-over entire data centers very often (have they ever?), so when it needed to happen, it didn't work. Anyway, I'm thinking this guy has only to back up his photo store about once a day to (something big) and put it in…

Amazon fails over data centers at least once a year in fire drill testing. Amazon has more backups, testing, and monitoring than 99.9% of small or midsized organizations.

Re: Why we built a 40TB photo server in-house instead of using S3

#70

Earlier quoted context omitted.

Sure it can, but can building and administering your own solution compete with a TCO less than S3's $0.14/GB with a 99.99% availability?

Of course it can, given how many useful open source solutions are available nowadays. Considering the total cost of 6000$ for a single server, you would be dropping 12k from the get-go for two servers (though you should really lease over a 12 month period if you can't afford it) and then you need to add colo costs for the machines, which can be around 300-400$ per machine in a single unit colocation environment, depe…

[deleted]
Post reply on HN