Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we built a 40TB photo server in-house instead of using S3

tech.oyster.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Why we built a 40TB photo server in-house instead of using S3

#41

Post by WettowelReactor »

dspillett wrote: post #30
mbell wrote: post #15

This is honestly pretty scary. There are a lot of single points of failure in this solution. 1) Single Box 2) Single Location 3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB) 4) Single bonded network link means single switch, no redundancy against switch / network device failure Honestly this may have been cheap but your getting what your paying for…

Warning, the sound of a broken record coming up... > An unreliable backup solution. Nope. RAID is not a backup solution . It provides redundancy so the array can survive an event like a device failure (and so the data survives as a consequence) without significant downtime for repair (with zero downtime if you have hot-swap hardware) but it does not, and is not intended to, protect the data from the huge list of othe…

Hot spares do not provide less downtime compared to a supported drive failure (ie. 2 disks on raid 6). they just reduce the mean time to repair. Handy if your disks are at a remote site or your dealing with very large arrays but likely a waste of spindles if your sitting next to your data all day.

Re: Why we built a 40TB photo server in-house instead of using S3

#42

Post by WettowelReactor »

fbuilesv wrote: post #3

For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).

Well first off all S3 is not the only option. Hell even going mid tier with an established SAN installed at a cheap colo is way cheaper than S3 and leaps ahead of their solution.

Re: Why we built a 40TB photo server in-house instead of using S3

#43

Post by JoeAltmaier »

jwatte wrote: post #6

S3 gives you multi-host, multi-region redundancy. Putting it all in one box is asking for trouble. What if the raid controller grows a bug and corrupts on write? It's happened. What if there's a fire in the building that has both your server and your back-up? Eggs, meet basket!

S3 fails too, right? Didn't they have an internal network issue this year, and go down for hours?

Anything you haven't tried, doesn't work. That's a truism in computing. I don't think Amazon tries failing-over entire data centers very often (have they ever?), so when it needed to happen, it didn't work.

Anyway, I'm thinking this guy has only to back up his photo store about once a day to (something big) and put it in his bank box, and he's good to go, at least for a photo site.

Re: Why we built a 40TB photo server in-house instead of using S3

#44

Post by meroliph »

res0nat0r wrote: post #35
meroliph wrote: post #27

Earlier quoted context omitted.

Building your own storage can provide the same redundancy and reliability. You can still use a CDN as well.

Sure it can, but can building and administering your own solution compete with a TCO less than S3's $0.14/GB with a 99.99% availability?

Of course it can, given how many useful open source solutions are available nowadays.

Considering the total cost of 6000$ for a single server, you would be dropping 12k from the get-go for two servers (though you should really lease over a 12 month period if you can't afford it) and then you need to add colo costs for the machines, which can be around 300-400$ per machine in a single unit colocation environment, depending on your location. This brings us to 21600$ in the first year for two machines colocated in separate datacenters, without any sysadmin costs;

The article states S3 would cost 60,000$ per year, and this doesn't include any bandwidth costs, whereas the colocation setup would include some decent bandwidth per month. Over time, it's easy to see how a lot of money can be saved.

Also, S3 is designed for 99.99% availability, but they only guarantee 99.9% through SLA, which isn't extremely spectacular. Hitting 99.9% isn't particularly hard if you have a sys admin worth his salt to set things up right, and with your own solution you can have more direct access to the file system as well as the ability to adhere to any regulations.

Re: Why we built a 40TB photo server in-house instead of using S3

#45

Post by buff-a »

baddox wrote: post #31
buff-a wrote: post #25

Earlier quoted context omitted.

Its just not that hard people! Assuming he built the box for $6000 as claimed (unfortunate timing given hard-drive prices), that's $54,000 of someone's time before it becomes a loss. I've got 7Tb sitting here and in the last two years I've had three failures (all seagate incidentally) for a total consumption of my time of 45 minutes and zero downtime. Maybe 10 hours (erring on the high side) to set the things up. If…

Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think o…

I'm paranoid, so I have two rules:

1. If you are using a hardware raid solution, then RAID1, and only if one of the drives can be pulled out and used by a standard SATA controller to get the data off. Controllers that munge the MBR just go in the trash.

2. Otherwise software raid1 or raid10.

Raid5 and Raid6 IMHO are more trouble than they are worth. Now, true, I'm not spending $6k on drives, so maybe at that quantity, Raid6 makes more sense. But there I would argue that the money you save on drives, you throw away as soon as it goes tits up. I've tried Raid5 with promise and areca cards and the performance was not as good as software raid10. Plus, on several occasions the transfer to the hot-spare failed and brought the array down. The data was there, but it required operator assistance (me). Until it went down, it was staggeringly slow.

If you really want to make sure that the array is available, use Raid1 with three or more drives. They are just so cheap. I was responsible for a windows server at a start-up a long while ago, and Raid5 sucked ass. Bonus: one day two drives failed. Tape restore is slow. Oh yeah: Don't just buy four drives and slap them in. Quite often a whole batch of drives will be bad. Buy drives from different manufacturers and beware that the sizes will be every so slightly different, so make your partitions smaller than the smallest.

However, all of this is irrelevant if you really want a lot of storage. For that it seems that a distributed, redundant file system would be best. If you want, you should be able to pull a drive or two out of every server, or all the drives out of a few servers, and still have all your data.

Re: Why we built a 40TB photo server in-house instead of using S3

#46

Post by PaulHoule »

mbell wrote: post #15

This is honestly pretty scary. There are a lot of single points of failure in this solution. 1) Single Box 2) Single Location 3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB) 4) Single bonded network link means single switch, no redundancy against switch / network device failure Honestly this may have been cheap but your getting what your paying for…

S3 is more expensive than cheap storage, but S3 provides a level of data protection that you'd have a hard time duplicating at any cost.

Re: Why we built a 40TB photo server in-house instead of using S3

#47

Post by Retric »

PaulHoule wrote: post #46
mbell wrote: post #15

This is honestly pretty scary. There are a lot of single points of failure in this solution. 1) Single Box 2) Single Location 3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB) 4) Single bonded network link means single switch, no redundancy against switch / network device failure Honestly this may have been cheap but your getting what your paying for…

S3 is more expensive than cheap storage, but S3 provides a level of data protection that you'd have a hard time duplicating at any cost.

In this case they already have a CDN which keeps 1 or more copy's of their data already and they just want a backup which makes S3 cost prohibitive. (You can save significant cash by using a service which gives much lower bandwidth access to that backup data.)

PS: Honestly, if their custom solution only add's one 9 of redundancy that may be fine. Their CDN is probably at 98+% up time or so which bumps them into the 99.8+ range which is better than most services out there.

Re: Why we built a 40TB photo server in-house instead of using S3

#48

Post by Keyframe »

thaumaturgy wrote: post #40
baddox wrote: post #31

Earlier quoted context omitted.

Out of curiosity, what's your 7TB setup? I've got about 10TB in about the worst setup imaginable: just a bunch of individual drives on a first-gen Atom running Windows Server 2008 (long story). I've been searching for a good home storage solution for a while, reading about various RAID cards and soft solutions like ZFS, but there's nothing that I'm willing to trust. I'm curious what your setup is and what you think o…

I've been doing basic RAID stuff for a while, most notably for a client that does physics modeling (and other things). So far, my favorite build is: - A box from abmx.com. Really nice people, good hardware. - An Areca RAID controller (might have to talk to them about this). - FreeBSD, which has good support for Areca controllers, thanks to Areca's friendliness towards the open source community. - ZFS. This particular…

Can you, or anyone else, give opinion about LSi 92xx MegaRAID cards vs ATTO r680 vs Areca? They all seem the same to me.

Re: Why we built a 40TB photo server in-house instead of using S3

#49

Post by jodrellblank »

mbell wrote: post #15

This is honestly pretty scary. There are a lot of single points of failure in this solution. 1) Single Box 2) Single Location 3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB) 4) Single bonded network link means single switch, no redundancy against switch / network device failure Honestly this may have been cheap but your getting what your paying for…

4) Single bonded network link means single switch, no redundancy against switch / network device failure

Maybe, but not necessarily. Could be a pair of stacked switches (making one logical switch), with the bonded link having one port on each physical switch.

Re: Why we built a 40TB photo server in-house instead of using S3

#50

Post by justincormack »

Retric wrote: post #47
PaulHoule wrote: post #46

Earlier quoted context omitted.

S3 is more expensive than cheap storage, but S3 provides a level of data protection that you'd have a hard time duplicating at any cost.

In this case they already have a CDN which keeps 1 or more copy's of their data already and they just want a backup which makes S3 cost prohibitive. (You can save significant cash by using a service which gives much lower bandwidth access to that backup data.) PS: Honestly, if their custom solution only add's one 9 of redundancy that may be fine. Their CDN is probably at 98+% up time or so which bumps them into the 9…

I can't find any documented evidence of data loss on S3. Can you?
Post reply on HN