Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we built a 40TB photo server in-house instead of using S3

tech.oyster.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Why we built a 40TB photo server in-house instead of using S3

#51

Post by jodrellblank »

JoeAltmaier wrote: post #43
jwatte wrote: post #6

S3 gives you multi-host, multi-region redundancy. Putting it all in one box is asking for trouble. What if the raid controller grows a bug and corrupts on write? It's happened. What if there's a fire in the building that has both your server and your back-up? Eggs, meet basket!

S3 fails too, right? Didn't they have an internal network issue this year, and go down for hours? Anything you haven't tried, doesn't work. That's a truism in computing. I don't think Amazon tries failing-over entire data centers very often (have they ever?), so when it needed to happen, it didn't work. Anyway, I'm thinking this guy has only to back up his photo store about once a day to (something big) and put it in…

Anything you haven't tried, doesn't work. That's a truism in computing.

(Aside) I wonder if this is more accurately stated: Anything you haven't tried recently, doesn't work. Even if you have tried it recently, that's no guarantee.

Or, to paraphrase Hofstadter's Law, the probability of something working some time after you last tried it drops surprisingly quickly, even when you take into account this rule.

Re: Why we built a 40TB photo server in-house instead of using S3

#52

Post by [deleted] »

justincormack wrote: post #50
Retric wrote: post #47

Earlier quoted context omitted.

In this case they already have a CDN which keeps 1 or more copy's of their data already and they just want a backup which makes S3 cost prohibitive. (You can save significant cash by using a service which gives much lower bandwidth access to that backup data.) PS: Honestly, if their custom solution only add's one 9 of redundancy that may be fine. Their CDN is probably at 98+% up time or so which bumps them into the 9…

I can't find any documented evidence of data loss on S3. Can you?

[deleted]

Re: Why we built a 40TB photo server in-house instead of using S3

#53

Post by spydum »

aaronjg wrote: post #10
fbuilesv wrote: post #3

For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).

Amazon S3 price also takes into account having redundancy at three data centers. So you should multiply the cost by 3, just for that. Of course if you don't need the redundancy, you can build it for cheaper. Also for some applications it really does make sense to move away from S3, and have a solution in house. For example The Broad Institute has about 6 petabytes of storage [1]. They in particular benefit from local…

Also quite important to mention, use of S3 will chew up your upstream bandwidth while making backups. Also, it will affect your recovery time objectives if you ever needed to run a restore (you will NOT be getting 200MByte/sec to or from S3, unless you have some sweet network connectivity).

Re: Why we built a 40TB photo server in-house instead of using S3

#54

Post by dspillett »

jemka wrote: post #38
dspillett wrote: post #30

Earlier quoted context omitted.

Warning, the sound of a broken record coming up... > An unreliable backup solution. Nope. RAID is not a backup solution . It provides redundancy so the array can survive an event like a device failure (and so the data survives as a consequence) without significant downtime for repair (with zero downtime if you have hot-swap hardware) but it does not, and is not intended to, protect the data from the huge list of othe…

>Nope. RAID is not a backup solution. OP isn't saying RAID is a backup solution. RAID is being used as a component within the backup solution (the server).

Ah, yes. Sorry.

My knee has been playing up and jerked a little there.

RAID is certainly a valuable part of many a more substantial backup solution.

Re: Why we built a 40TB photo server in-house instead of using S3

#55

Post by 16s »

A bit off-topic, but I wonder if anyone can comment on using software RAID rather than hardware RAID? I'd like to try it. I've been bitten by buggy hardware RAID controllers far too often (even high-dollar name brand gear). I know that all of the free *nix systems offer software RAID, I'm just curious how they perform.

Re: Why we built a 40TB photo server in-house instead of using S3

#56

Post by dspillett »

WettowelReactor wrote: post #41
dspillett wrote: post #30

Earlier quoted context omitted.

Warning, the sound of a broken record coming up... > An unreliable backup solution. Nope. RAID is not a backup solution . It provides redundancy so the array can survive an event like a device failure (and so the data survives as a consequence) without significant downtime for repair (with zero downtime if you have hot-swap hardware) but it does not, and is not intended to, protect the data from the huge list of othe…

Hot spares do not provide less downtime compared to a supported drive failure (ie. 2 disks on raid 6). they just reduce the mean time to repair. Handy if your disks are at a remote site or your dealing with very large arrays but likely a waste of spindles if your sitting next to your data all day.

Not hot spares (which you are right, don't reduce array repare downtime): hot swap (which can). If a hot spare gets used you still need to swap out the broken device (its replacement becoming a new hot spare).

Wether you are swapping out a replacement for an active drive or swapping out a dead drive with what will be a new hot spare (a drive that was previously the spare now being an active drive in teh array) whether you have hot-swap or not makes a difference. Without, you have to power down the array for a short time which in an environment that needs availability to be as high as possible could be an issue.

Re: Why we built a 40TB photo server in-house instead of using S3

#57

Post by Retric »

justincormack wrote: post #50
Retric wrote: post #47

Earlier quoted context omitted.

In this case they already have a CDN which keeps 1 or more copy's of their data already and they just want a backup which makes S3 cost prohibitive. (You can save significant cash by using a service which gives much lower bandwidth access to that backup data.) PS: Honestly, if their custom solution only add's one 9 of redundancy that may be fine. Their CDN is probably at 98+% up time or so which bumps them into the 9…

I can't find any documented evidence of data loss on S3. Can you?

S3 has lost data 3 times in its history.

Re: Why we built a 40TB photo server in-house instead of using S3

#58

Post by mbell »

jodrellblank wrote: post #49
mbell wrote: post #15

This is honestly pretty scary. There are a lot of single points of failure in this solution. 1) Single Box 2) Single Location 3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB) 4) Single bonded network link means single switch, no redundancy against switch / network device failure Honestly this may have been cheap but your getting what your paying for…

4) Single bonded network link means single switch, no redundancy against switch / network device failure Maybe, but not necessarily. Could be a pair of stacked switches (making one logical switch), with the bonded link having one port on each physical switch.

Good point. However, given the lack of redundancy elsewhere, and the fact that a pair of such switches could approach the cost of this server, I'd be willing to bet that this isn't the case.

Re: Why we built a 40TB photo server in-house instead of using S3

#59

Post by cbs »

jackowayed wrote: post #21

> In strict accordance with KISS methodology Buying a ton of parts, carefully assembling them, and having it be your problem when something breaks is simpler than paying Amazon to solve the problem nearly perfectly?

If you know what you're doing, yeah. Running a fileserver should be pretty damn simple for anyone who stylized themselves a "hacker", but you're talking like maintaining file servers is some sort of black magic for which the simple solution is to give up and turn to an outsourced solution where you have little control and no visibility.

Its not. The hard part of storage is to understand the capabilities and limitations of your storage and more importantly how those fit in with your computational needs. You have to do that at amazon on in house. A3 just makes it easy to ignore evaluating their service because you're never actually forced to. Maintaining servers is butter.

Re: Why we built a 40TB photo server in-house instead of using S3

#60

Post by mbell »

16s wrote: post #55

A bit off-topic, but I wonder if anyone can comment on using software RAID rather than hardware RAID? I'd like to try it. I've been bitten by buggy hardware RAID controllers far too often (even high-dollar name brand gear). I know that all of the free *nix systems offer software RAID, I'm just curious how they perform.

I can't comment for industrial usage but for use at home (work and non-work use) I have the follow setup for storage:

My old desktop hardware (Intel Q6600, 8GB RAM, Asus MB, pair of gigabit links bonded)

Supermicro 4u tower case with 8x hot swap bays + the 5.25 bay filled with a 5x hot swap cage, 13 total hot swap bays.

3-Ware 9550SX RAID Controller, 4 x WB RE 320GB drives, RAID 5.

8 x 1.5TB "Green" Drives, mixture of WD and Samsung drives, RAID 6 using mdadm (linux software raid)

1 x WD Raptor (system drive)

Ubuntu with KVM for virtutalization

Originally i was in the "must have raid controller" camp which is when I bought the 3-ware controller and the RE series drives. When that array filled up I did some research and decided to just go the mdadm route and have minimal complaints so far. I use the 3-ware array still for "critical data" and back it up to S3.

Monitoring: You have to work a bit more to get proper alerting of issues from mdadm but its not hard to setup. Doesn't matter for me much, i sit in the same room as this server most of the day working so if something goes wrong i generally notice before i get the e-mail.

Performance: As mentioned I'm using Green drives, this system wasn't built with speed in mind but rather large amounts of nearline storage. Never-the-less, with some basic tweaking and making sure the array's partition alignment is correct I get around 450MB/sec read speed and ~85MB/sec write long term, I have the system setup to cache writes aggressively however as most of the data on this array isn't critical and its on a UPS, what this means is that writes under a few gigabytes usually complete at wire speed (~200MB/sec) then get flushed to the disk later. Most of the time I'm limited by network bandwidth to this system, unless I'm writing a very large amount of data all at once.

One negative is rebuild speed, here i'm very limited by the 'Green' drives I believe. It runs at about 50MB/sec so rebuilds do take awhile.

As far as CPU usage goes, I've never seen it be the limiter but haven't watched it that closely, it doesn't peg during rebuild. This machine acts as an SMB/NFS file server and a few development VMs 24/7 (database, and a couple other things) and I've never really had an issue with cpu usage.

One really nice bonus is that if something in the system fails, you can just plug the drives of your array into to almost any other linux system:

apt-get install mdadm

mdadm --assemble --scan

Poof, working array.

tl;dr

If your going after 1GB/sec transfer speeds, get a high end raid card.

If you just need some large redundant storage that can saturate a Gbit link or 2, then mdadm software RAID is just fine IMO.

Post reply on HN