Live data from Hacker News

“Let's talk about a hypothetical public-facing service”

reddit.com

61–70 of 72 posts

Re: “Let's talk about a hypothetical public-facing service”

#61
post #21

...Extremely Massive Corporation... I knew EMC storage was utter shit when, upon attempting to create a new RAID group, I realized that the configuration tool's default was to stripe across drives within a shelf , not to create stripes that span shelves . Worse, to create the more fault-tolerant, shelf-spanning RAID volumes, one must manually add drives, one by one to the array, in a process that involves about 44 (s…

Actually it sounds like the EMC software was trying to save you from something which is a bad practice. If you want raid 1+0 it is better to have two disks on the same controller/chassis in raid 0, then raid 1 (or raid 5/6) across chassis. Otherwise a chassis failure will take fail both leaves of your raid mirror.

Having said that, EMC is stupidly overpriced bloat.

Re: “Let's talk about a hypothetical public-facing service”

#62
post #22

Earlier quoted context omitted.

Sadly, depending on what you want to accomplish with your storage the future is still not here. For example, if you want to share a virtual disk with multiple servers through NFS you will still need to manually create a NFS high-availability cluster using e.g. DRBD on most cloud providers (e.g. EC2 or Azure). To be fair, MS Azure offers fully managed SMB shares as a service, but the transfer speed is quite low (60 MB…

AWS has EFS [1] which should allow you to share one FS to multiple serves. It is still in preview but could help address these needs. [1] http://aws.amazon.com/efs/pricing/

That looks interesting, thanks for the link!

Re: “Let's talk about a hypothetical public-facing service”

#64
post #15

Earlier quoted context omitted.

Come on, read Extremely Massive Corporation again, and try to think of an Extremely Massive Corporation which happens to deal in computer storage systems.

Some of us aren't in the environment where we'd know who EMC are :-)

Believe me, you are happier for it!

Re: “Let's talk about a hypothetical public-facing service”

#65
> It writes a full 32 bits of numeric user ID to its filesystem, but to save a few bytes it only stores 16 bits of group IDs. Some engineer probably thought that'd be enough for anybody.

I'm having the same issue with the number of hardlinks, which, for linux ext4 systems, is limited to 65000.

Re: “Let's talk about a hypothetical public-facing service”

#66

SourceForge has posted info about current infrastructure and service restoration activity at http://sourceforge.net/blog/sourceforge-infrastructure-and-s... [Disclosure: I work for SourceForge]

>[Disclosure: I work for SourceForge]

My condolences.

Re: “Let's talk about a hypothetical public-facing service”

#67
post #21

...Extremely Massive Corporation... I knew EMC storage was utter shit when, upon attempting to create a new RAID group, I realized that the configuration tool's default was to stripe across drives within a shelf , not to create stripes that span shelves . Worse, to create the more fault-tolerant, shelf-spanning RAID volumes, one must manually add drives, one by one to the array, in a process that involves about 44 (s…

Actually it sounds like the EMC software was trying to save you from something which is a bad practice. If you want raid 1+0 it is better to have two disks on the same controller/chassis in raid 0, then raid 1 (or raid 5/6) across chassis. Otherwise a chassis failure will take fail both leaves of your raid mirror. Having said that, EMC is stupidly overpriced bloat.

Did I really need to spell out the geometry of the RAID I was creating to avoid this kind of nitpicky follow-up?

It was a 20-something disk RAID 10 [1], arranged so that every mirrored pair of disks spanned different enclosures, in order to mitigate the failure of any one shelf — that is, interleaving mirrors across controllers and shelves, exactly as you suggest I should have done — and further, such that any one shelf failing only affects the mirrors that had disks on that shelf.

EMC's software wanted to allocate the drives from two shelves, with an unequal number of drives per shelf. It was just grabbing the next however many disks, linearly.

So, no, they weren't trying to balance the mirror across enclosures or controllers. They just weren't thinking.

[1] By "RAID 10", I mean "striped mirrors" — that is, create a bunch of mirrors that span shelves and then stripe across them — not "mirrored stripes" which is what you appear to be suggesting, with "it is better to have two disks on the same controller/chassis in raid 0, then raid 1".

A striped mirror is recommended in everything I've ever read on the subject, because it puts the redundancy at the lowest level of the array's geometry.

Using a mirrored stripe, on the other hand, means that when one disk fails, any other disks striped with it, still presumably perfectly functional, can't be used; the controller must instead read from and write to the mirror. If a disk in that mirror subsequently fails, you've lost data — and remember that when striping, the chance of failure is multiplied by the number of disks in the stripe.

EDIT: Footnote.

Re: “Let's talk about a hypothetical public-facing service”

#68

Earlier quoted context omitted.

The TL;DR answer is yes, there are some hard drive SMART values that can indicate failure is likely, but they vary by model and don't necessarily show before failure.

yeah, I was wondering if there were measurables that could be correlated with failure before SMART kicked in, even if they were something like date of year, or location of manufacture, or shipping route they took. :P

Generally the most common measurable is sector reallocation errors. This comes from various random things going wrong at the wrong time and the disk re-allocates a new sector from the spare pool to deal with one that has gone bad. In operation, our disks at Blekko pick up sector reallocation errors at a low statistical rate that picks up prior to total failure. Since our infrastructure is triply redundant (three disks hold a copy of every piece of data) we can simply reformat drives which develop sector errors. If you plot the time between sector errors developing over the life of the drive, it gets shorter rapidly as the drive as nearing complete failure. Sometimes however there is no warning, the drive simply fails. As with my previous experience at Google and NetApp before that, there is a small rise in early failure (infant mortality) then a long tail toward a steep failure rate after about 10 years.

Re: “Let's talk about a hypothetical public-facing service”

#69
post #67

Earlier quoted context omitted.

Actually it sounds like the EMC software was trying to save you from something which is a bad practice. If you want raid 1+0 it is better to have two disks on the same controller/chassis in raid 0, then raid 1 (or raid 5/6) across chassis. Otherwise a chassis failure will take fail both leaves of your raid mirror. Having said that, EMC is stupidly overpriced bloat.

Did I really need to spell out the geometry of the RAID I was creating to avoid this kind of nitpicky follow-up? It was a 20-something disk RAID 10 [1], arranged so that every mirrored pair of disks spanned different enclosures, in order to mitigate the failure of any one shelf — that is, interleaving mirrors across controllers and shelves, exactly as you suggest I should have done — and further, such that any one sh…

Don't get all upset. You said stripe disks in your first comment, not mirror volumes.

In a big system I am usually more concerned with avoiding any SPOF (that will cause downtime), then adding redundancy for the most likely failures.

Individual mirrored pairs on split chassis will mitigate against long rebuild times for a single disk failure but it requires an all software architecture as the individual HBA can't do hot spare rebuild. It also reduces the benefit of the HBA cache. So intra chassis, striped mirrors; inter chassis, mirrored stripes. Of course, mirroring across striped mirrors would be better, but the cost to peak write and capacity might rule that out.

Also, yes I believe you, EMC sucks. But they do have some good engineers.

Re: “Let's talk about a hypothetical public-facing service”

#70

Earlier quoted context omitted.

yeah, I was wondering if there were measurables that could be correlated with failure before SMART kicked in, even if they were something like date of year, or location of manufacture, or shipping route they took. :P

Generally the most common measurable is sector reallocation errors. This comes from various random things going wrong at the wrong time and the disk re-allocates a new sector from the spare pool to deal with one that has gone bad. In operation, our disks at Blekko pick up sector reallocation errors at a low statistical rate that picks up prior to total failure. Since our infrastructure is triply redundant (three disk…

Thanks a bunch for this.
Post reply on HN