Why we built a 40TB photo server in-house instead of using S3

> In strict accordance with KISS methodology

Buying a ton of parts, carefully assembling them, and having it be your problem when something breaks is simpler than paying Amazon to solve the problem nearly perfectly?

"The one most valuable asset at Oyster.com is our photo collection. ... In strict accordance with KISS methodology, we opted against LTO and S3, and decided to build a big BOX."

I can only imagine how much they will be scared each time they need install updates or reboot THE BOX. They will eventually decide to build identical BOX and mirror their data on daily basis. Then they will notice that mirroring such big volumes of data is wasting tooo much system resources and start evaluating in-house distributed storage solutions, such as OpenStack Swift. Then they will notice it is way too overcomplicated and finally decide migrate their data to Amazon S3.

I'm writing it as a person who walked the same path over the last few years. :-)

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up.

If I needed this much space in a start-up I'd totally do it myself - it would be a good use of my time. No, my concern with this setup is that it looks (to me) to be a "w00t I got $6000 so lets build a 733t boxen!" It looks cool and fragile, instead of boring and robust.

UPDATE: To be fair, I'm not hammering those drives continuously. How much does that change the equation? Even if it goes from 45 minutes every two years to 45 minutes per day, you'd still be net positive. Basically, you can't prevent drive failures. But if you can keep the failures restricted to the ones that just require popping in a new drive, then your "operator time" is minimal. Its when you lose an array that you're in trouble. Again: Do not use one big box!

Building your own storage can provide the same redundancy and reliability. You can still use a CDN as well.

For a 24 drive server, I'd just get a heavily discounted dell or HP box. A startup should be able to pay half list, buy two, and be ahead vs. s3.

Supermicro chassis re a big improvement overdoing your own wiring. The Areca controllers, especially in raid 6, re great. For raid 5 I'd also look at 3ware.

For single gig e, you can get away with esata expanders, building something like the back blaze pod. I've done that kind of thing for personal use, and to have an onsite mirror of something, but I'd want several, in several different colo facilities, to compare with s3. The exception is if you need some kind of scratch storage, but even refilling a 40TB archive with downloadable content takes a really long time over a 1Gbps link.

I'd build a few boxes like this now, but the Thai floods pushed hard drive prices up to the point I have to wait. Hopefully fast 4-5TB drives will be $200 by summer 2012.