Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Let's talk about a hypothetical public-facing service”

reddit.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: “Let's talk about a hypothetical public-facing service”

#51

Post by ObviousScience »

snowwrestler wrote: post #44
ObviousScience wrote: post #41

Earlier quoted context omitted.

I don't believe maintaining a high availability system with proper checking is less than 3 hours a day of work, or ~10% of someone's time on duty if we're talking around the clock. So we're talking about $75,000-90,000 a year in salary to maintaining the cluster if you want 24x365 coverage (which Amazon provides), as you'll need 2 people per shift, 3 shifts per day at a minimum to have people in house, even if they'r…

I feel like these comparisons oversell the level of support that actually comes with Amazon. Yes AWS as a whole very rarely goes down, but instances have problems all the time, and who do you call? When you have your own servers and staff, even if they are just on-call with a pager, you know they are going to work for you on your problem until it is fixed. In comparison, the sentiment about AWS is this: better build…

This is simply a perspective on systems engineering.

My argument would be that part of the system is always down, the question is which part and how long, and how that impacts the system performance on the whole.

AWS services work well if you build a stateless system which is in some senses "embarrassingly parallelizable", because you can talk about the capacity of such a system, and the impact of non-functioning components is easy to predict. This is how most standard engineering is done, across disciplines.

Traditionally, it has not been the case in computer systems, but most modern techniques advocate using such systems, because they're MUCH more reliable.

You just sound like you need a safety blanket for emotional reasons, not that you're making sound engineering points about how to most cheaply engineer a high availability system.

I mean, do you really believe AWS engineers aren't working hard to keep their system fully functional?

Re: “Let's talk about a hypothetical public-facing service”

#52

Post by maccam94 »

codezero wrote: post #32
ChuckMcM wrote: post #30

Earlier quoted context omitted.

AFR buddy, AFR. AFR is the annual failure rate, its typically 2 - 5% of the population per year. 4 drives you don't have a large enough set to see this in action, just every day you're in danger of losing a drive by a small statistical amount. In the Blekko cluster we have just under 10,000 drives. We have a two 20 drive 'boxes' (40 drives) from Western Digital, as drives fail we pull replacements from the 'new/refur…

Just out of curiosity, has anyone done any research into determining whether a drive which fails after X days/years has some properties in the first Y days that could be a signal for future failure?

The TL;DR answer is yes, there are some hard drive SMART values that can indicate failure is likely, but they vary by model and don't necessarily show before failure.

Re: “Let's talk about a hypothetical public-facing service”

#53

Post by Veratyr »

nadams wrote: post #46
ChuckMcM wrote: post #30

Earlier quoted context omitted.

AFR buddy, AFR. AFR is the annual failure rate, its typically 2 - 5% of the population per year. 4 drives you don't have a large enough set to see this in action, just every day you're in danger of losing a drive by a small statistical amount. In the Blekko cluster we have just under 10,000 drives. We have a two 20 drive 'boxes' (40 drives) from Western Digital, as drives fail we pull replacements from the 'new/refur…

> just every day you're in danger of losing a drive by a small statistical amount. Which is why for important data I always use some sort of RAID (or cloud syncing). If I lose a drive I won't lose all my data (presumably though if I bought both drives at the same time there is a chance that both could fail at the same or close to the same time). > That said, if you're running your ReadyNAS with raid 10 (mirrored driv…

> Which is why for important data I always use some sort of RAID (or cloud syncing). If I lose a drive I won't lose all my data

Just a friendly reminder that RAID != backup. There are numerous data loss cases that RAID does not deal with.

Personally I use striped ZFS with important volumes periodically snapshotted, replicated to external (and encrypted) disks and then stored offsite (cycle through a couple of sets of disks). Most important data is also periodically synced to cloud storage (as well as offsite disk).

This accounts for:

- Single disk failure (striping)

- Bitrot (ZFS scrubbing can reveal bitrot on disk and correct it from parity)

- Human error (snapshots)

- Catastrophic damage to home NAS (offsite backups)

RAID alone (depending on the particular implementation) will generally not deal with the 3 latter failure cases.

Re: “Let's talk about a hypothetical public-facing service”

#55

Post by jks »

codezero wrote: post #32
ChuckMcM wrote: post #30

Earlier quoted context omitted.

AFR buddy, AFR. AFR is the annual failure rate, its typically 2 - 5% of the population per year. 4 drives you don't have a large enough set to see this in action, just every day you're in danger of losing a drive by a small statistical amount. In the Blekko cluster we have just under 10,000 drives. We have a two 20 drive 'boxes' (40 drives) from Western Digital, as drives fail we pull replacements from the 'new/refur…

Just out of curiosity, has anyone done any research into determining whether a drive which fails after X days/years has some properties in the first Y days that could be a signal for future failure?

Yes: https://www.backblaze.com/blog/hard-drive-smart-stats/

Re: “Let's talk about a hypothetical public-facing service”

#56

Post by microcolonel »

pjc50 wrote: post #17
aidos wrote: post #15

Earlier quoted context omitted.

Some of us aren't in the environment where we'd know who EMC are :-)

There was a time when they had adverts on seemingly every page of every computing magazine. Although that may have been back when magazines were a thing and Sun were a hardware company.

I'm too young to drink, I figured this reference would work just fine on the GP here... Though the five downvotes I got on my message tell a different story. :-P

Re: “Let's talk about a hypothetical public-facing service”

#57

Post by codezero »

maccam94 wrote: post #52
codezero wrote: post #32

Earlier quoted context omitted.

Just out of curiosity, has anyone done any research into determining whether a drive which fails after X days/years has some properties in the first Y days that could be a signal for future failure?

The TL;DR answer is yes, there are some hard drive SMART values that can indicate failure is likely, but they vary by model and don't necessarily show before failure.

yeah, I was wondering if there were measurables that could be correlated with failure before SMART kicked in, even if they were something like date of year, or location of manufacture, or shipping route they took. :P

Re: “Let's talk about a hypothetical public-facing service”

#58

Post by kragen »

rosser wrote: post #21

...Extremely Massive Corporation... I knew EMC storage was utter shit when, upon attempting to create a new RAID group, I realized that the configuration tool's default was to stripe across drives within a shelf , not to create stripes that span shelves . Worse, to create the more fault-tolerant, shelf-spanning RAID volumes, one must manually add drives, one by one to the array, in a process that involves about 44 (s…

I guess a "shelf" shares some piece of hardware, such as a controller card or communication backplane, which could itself fail and thus disable access to all the drives on that shelf, which a RAID 4 or 5 stripe that spans shelves could survive?

Re: “Let's talk about a hypothetical public-facing service”

#59

Post by [deleted] »

ObviousScience wrote: post #51
snowwrestler wrote: post #44

Earlier quoted context omitted.

I feel like these comparisons oversell the level of support that actually comes with Amazon. Yes AWS as a whole very rarely goes down, but instances have problems all the time, and who do you call? When you have your own servers and staff, even if they are just on-call with a pager, you know they are going to work for you on your problem until it is fixed. In comparison, the sentiment about AWS is this: better build…

This is simply a perspective on systems engineering. My argument would be that part of the system is always down, the question is which part and how long, and how that impacts the system performance on the whole. AWS services work well if you build a stateless system which is in some senses "embarrassingly parallelizable", because you can talk about the capacity of such a system, and the impact of non-functioning com…

[deleted]
Post reply on HN