Why we built a 40TB photo server in-house instead of using S3 • Hacker News Board

Re: Why we built a 40TB photo server in-house instead of using S3

#71

Post by gujk » Fri, Dec 23, 2011, 9:11 PM UTC

meroliph wrote: post #44

res0nat0r wrote: post #35

Earlier quoted context omitted.

Sure it can, but can building and administering your own solution compete with a TCO less than S3's $0.14/GB with a 99.99% availability?

Of course it can, given how many useful open source solutions are available nowadays. Considering the total cost of 6000$ for a single server, you would be dropping 12k from the get-go for two servers (though you should really lease over a 12 month period if you can't afford it) and then you need to add colo costs for the machines, which can be around 300-400$ per machine in a single unit colocation environment, depe…

Availability is an irrelevant metric for bulk storage. Durability matters.

Also, less than 99.9% of sysadmins are worth their salt, are you sure yours is?

Re: Why we built a 40TB photo server in-house instead of using S3

#72

Post by Jugglernaut » Sat, Dec 24, 2011, 12:00 AM UTC

gujk wrote: post #68

rorrr wrote: post #64

Earlier quoted context omitted.

That single box is made of replaceable cheap parts. Only a fire can take it completely out.

My company's data center has suffered a fire at least once in the past 5 years.

That sounds like a terrible data center. If you count a server short circuit with some minor smoke as a fire I guess it's understandable. The server room should have Halon/Inergen or a similar system with smoke detectors.

Building your own solution to beat S3 is certainly viable but at this scale I doubt it.

Re: Why we built a 40TB photo server in-house instead of using S3

#73

Post by Jugglernaut » Sat, Dec 24, 2011, 12:03 AM UTC

vilda wrote: post #39

smerritt wrote: post #17

We had something similar but smaller (~8 TB) at a place I worked, and it was a nightmare. Migrating from that to S3 was one of the best things to happen to that project. Being a single big box, it had a bunch of single points of failure, and boy did they fail; we probably had 5-10 hours per month of downtime due to the photo server falling over (flaky RAID controller firmware, mostly). Also, since the big box was exp…

This sounds like you had a really bad implementation. Proper file server of this small size would not fail for several hours per month.

I agree, that just should not happen unless the server was a complete lemon or badly assembled.

Re: Why we built a 40TB photo server in-house instead of using S3

#74

Post by Jugglernaut » Sat, Dec 24, 2011, 12:08 AM UTC

dfrankow wrote: post #5

How much more expensive is a clustered software solution (e.g., Hadoop FS) than this RAID box?

That depends on how much hardware you want to spread it out on. Many companies would have to hire a hadoop guy, then again some companies would have to hire a sysadmin to to run the Oyster solution. Build your storage to fit your company.

Re: Why we built a 40TB photo server in-house instead of using S3

#75

Post by smerritt » Sat, Dec 24, 2011, 1:41 AM UTC

vilda wrote: post #39

smerritt wrote: post #17

We had something similar but smaller (~8 TB) at a place I worked, and it was a nightmare. Migrating from that to S3 was one of the best things to happen to that project. Being a single big box, it had a bunch of single points of failure, and boy did they fail; we probably had 5-10 hours per month of downtime due to the photo server falling over (flaky RAID controller firmware, mostly). Also, since the big box was exp…

This sounds like you had a really bad implementation. Proper file server of this small size would not fail for several hours per month.

Absolutely true. The RAID controller would randomly lose drives and the driver for it would randomly cause kernel panics. We tried different firmwares and different kernels and made some progress, but never really got it stable under load.

However, that's the risk you run with single points of failure. Put all your data on one big box, and any failure in your RAID hardware, RAID firmware, RAID drivers, network drivers, kernel, RAM, OS, et cetera will take down the big box and thus take down anything relying on it.

The lesson I learned wasn't to make a super-robust single system, it was to have enough redundancy to stay up when something inevitably fails.

Re: Why we built a 40TB photo server in-house instead of using S3

#76

Post by WettowelReactor » Sat, Dec 24, 2011, 1:46 PM UTC

dspillett wrote: post #56

WettowelReactor wrote: post #41

Earlier quoted context omitted.

Hot spares do not provide less downtime compared to a supported drive failure (ie. 2 disks on raid 6). they just reduce the mean time to repair. Handy if your disks are at a remote site or your dealing with very large arrays but likely a waste of spindles if your sitting next to your data all day.

Not hot spares (which you are right, don't reduce array repare downtime): hot swap (which can). If a hot spare gets used you still need to swap out the broken device (its replacement becoming a new hot spare). Wether you are swapping out a replacement for an active drive or swapping out a dead drive with what will be a new hot spare (a drive that was previously the spare now being an active drive in teh array) whethe…

Sorry I misread your post. I just couldn't fathom a storage solution without hot swappable drives.

Re: Why we built a 40TB photo server in-house instead of using S3

#77

Post by rhizome » Sat, Dec 24, 2011, 10:54 PM UTC

smerritt wrote: post #17

We had something similar but smaller (~8 TB) at a place I worked, and it was a nightmare. Migrating from that to S3 was one of the best things to happen to that project. Being a single big box, it had a bunch of single points of failure, and boy did they fail; we probably had 5-10 hours per month of downtime due to the photo server falling over (flaky RAID controller firmware, mostly). Also, since the big box was exp…

flaky RAID controller firmware

Fortunately, this can usually be rectified with a simple application of money. Good hardware is its own reward.

However, it sounds like the problem that S3 cured was caused by a bad architecture.

Re: Why we built a 40TB photo server in-house instead of using S3

#78

Post by presto8 » Sat, Dec 24, 2011, 11:31 PM UTC

baddox wrote: post #31

buff-a wrote: post #25

Earlier quoted context omitted.

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If…

Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think o…

I have an encrypted 5TB setup using LVM, dm-crypt, and XFS in two identical full-size ATX 8-bay chassis, with nightly rsyncs from the primary to the secondary (with a simple versioning script that I wrote). Every year, I buy the biggest hard drive I can for $100 and replace the oldest drive in each system. LVM makes it extremely easy to move all of the blocks off of a drive and remove it. I also run smartd on both systems and replace a drive at the first sign of a serious error.

I used to run RAID hardware cards but changed to software-only solutions after a hardware card crashed. Unless you are going to buy two of the exact same RAID card (one as a hotspare), or unless performance is a big deal (which it shouldn't be on a backup system I would think), software-only solutions are the way to go.

I complement this with off-site continous backups of irreplaceable data (about 200GB) to CrashPlan.

I really wanted to use ZFS, but the Linux support isn't that great, and also I don't think it's possible to remove drives from ZFS arrays. Both of those were deal killers for me. In any event, LVM+XFS has worked out great. XFS is a very stable file system and has given me no trouble.