Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we built a 40TB photo server in-house instead of using S3

tech.oyster.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Why we built a 40TB photo server in-house instead of using S3

#31

Post by baddox »

buff-a wrote: post #25
fbuilesv wrote: post #3

For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If…

Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think of it.

Re: Why we built a 40TB photo server in-house instead of using S3

#32

Post by amadvance »

baddox wrote: post #31
buff-a wrote: post #25

Earlier quoted context omitted.

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If…

Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think o…

You can evaluate SnapRAID. It's a software, not realtime, RAID5/6 like solution. http://snapraid.sourceforge.net/

Re: Why we built a 40TB photo server in-house instead of using S3

#33

Post by res0nat0r »

rorrr wrote: post #22
jackowayed wrote: post #21

> In strict accordance with KISS methodology Buying a ton of parts, carefully assembling them, and having it be your problem when something breaks is simpler than paying Amazon to solve the problem nearly perfectly?

Better than going bankrupt.

You mean, better than going bankrupt until they go bankrupt when the single BOX goes down, right?

Re: Why we built a 40TB photo server in-house instead of using S3

#34

Post by lucaspiller »

cagenut wrote: post #11

Yes S3 can get expensive, but imho this swings the pendulum too far in the other direction. Something like a riak cluster of four 2U/24-drive servers would get you the cost structure of good colo but the features/resilience/operational-flexibility of something more like s3.

Good suggestion. I posted on the Riak mailing list regarding this:

http://lists.basho.com/pipermail/riak-users_lists.basho.com/...

Re: Why we built a 40TB photo server in-house instead of using S3

#35

Post by res0nat0r »

meroliph wrote: post #27
nodesocket wrote: post #7

Static image storage makes the most sense todo on S3 or similar. Building your own storage, does not provide the redundancy and reliability of S3. Additionally, you have the flexibility to enable CloudFront and distribute the images via CDN if you need.

Building your own storage can provide the same redundancy and reliability. You can still use a CDN as well.

Sure it can, but can building and administering your own solution compete with a TCO less than S3's $0.14/GB with a 99.99% availability?

Re: Why we built a 40TB photo server in-house instead of using S3

#36

Post by throwaway64 »

res0nat0r wrote: post #33
rorrr wrote: post #22

Earlier quoted context omitted.

Better than going bankrupt.

You mean, better than going bankrupt until they go bankrupt when the single BOX goes down, right?

why would they be limited to a single box?

Re: Why we built a 40TB photo server in-house instead of using S3

#37

Post by zorg »

"Having spent some extra time on research, fine-tuning, and optimizing the new server, we were glad to find that the gigabit network had became the bottleneck"

This matches my experience, RAID performance in linear IO is an order of magnitude below what the disk should allow. This guy is relieved to finally get 1 gig of useful bandwidth out of 24 disks (about 1 gig a piece). So it's no faster than a single disk.

(i know it's linear io in this case because the screenshot shows a large file copy)

Re: Why we built a 40TB photo server in-house instead of using S3

#38

Post by jemka »

dspillett wrote: post #30
mbell wrote: post #15

This is honestly pretty scary. There are a lot of single points of failure in this solution. 1) Single Box 2) Single Location 3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB) 4) Single bonded network link means single switch, no redundancy against switch / network device failure Honestly this may have been cheap but your getting what your paying for…

Warning, the sound of a broken record coming up... > An unreliable backup solution. Nope. RAID is not a backup solution . It provides redundancy so the array can survive an event like a device failure (and so the data survives as a consequence) without significant downtime for repair (with zero downtime if you have hot-swap hardware) but it does not, and is not intended to, protect the data from the huge list of othe…

>Nope. RAID is not a backup solution.

OP isn't saying RAID is a backup solution. RAID is being used as a component within the backup solution (the server).

Re: Why we built a 40TB photo server in-house instead of using S3

#39

Post by vilda »

smerritt wrote: post #17

We had something similar but smaller (~8 TB) at a place I worked, and it was a nightmare. Migrating from that to S3 was one of the best things to happen to that project. Being a single big box, it had a bunch of single points of failure, and boy did they fail; we probably had 5-10 hours per month of downtime due to the photo server falling over (flaky RAID controller firmware, mostly). Also, since the big box was exp…

This sounds like you had a really bad implementation. Proper file server of this small size would not fail for several hours per month.

Re: Why we built a 40TB photo server in-house instead of using S3

#40

Post by thaumaturgy »

baddox wrote: post #31
buff-a wrote: post #25

Earlier quoted context omitted.

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If…

Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think o…

I've been doing basic RAID stuff for a while, most notably for a client that does physics modeling (and other things). So far, my favorite build is:

- A box from abmx.com. Really nice people, good hardware.

- An Areca RAID controller (might have to talk to them about this).

- FreeBSD, which has good support for Areca controllers, thanks to Areca's friendliness towards the open source community.

- ZFS.

This particular box has had almost no downtime in the past year, and the downtime it's had has been caused either by administration tasks (software updates or configuration modification) or a USB flash drive failure (we've been experimenting with BSD and Linux server configurations on flash media separate from the server's data storage, so that we can do things like show up with replacement or upgraded configurations, plug them in, and be done. USB flash media is not reliable enough for this though, even if you're not doing swap on it).

I just added a couple of 3TB Hitachi drives to the array last week. Resizing the volume was a little fiddly because the Areca firmware needed an update first before it could recognize the 3TB drives (not a problem you'd have on newer models), but otherwise, everything happened live, while the box was up and running and doing its job. When the Areca controller software finished resizing the volume, zfs happily said, "oh, I'm bigger now! I can handle that!"

OpenBSD's softraid is also pretty good stuff, but I don't think it makes sense to use it unless you're trying to cheap out with a small 2-bay box and no controller in a RAID 1 or something.

SBS 2008 is not my favorite thing in the world.

Post reply on HN