Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Should I buy ECC memory? (2015)

danluu.com

Post reply on HN
Print view View original

71–80 of 237 posts

Re: Should I buy ECC memory? (2015)

#71

Post by blackflame7000 »

Dylan16807 wrote: post #63
dogma1138 wrote: post #40

Earlier quoted context omitted.

Because with ZFS bit rot can be cumulative, with most file systems a memory error will corrupt a file if the format can't handle errors, with ZFS overtime the entire volume can get corrupted especially when you are doing recovery or expansion, even in normal operation data is moved around quite a bit. For them most part with other common file systems when a file is written it stays there even in RAID.

I don't know, I wouldn't say ZFS moves files around any more than a typical filesystem.

Correct. ZFS won't do unnecessary file reordering unless a scrub has been initiated.

Re: Should I buy ECC memory? (2015)

#72

Post by loeg »

sytelus wrote: post #55
nostrademons wrote: post #7

While I was at Google, someone asked one of the very early Googlers (I think it was Craig Silverstein, but it may've been Jeff Dean) what was the biggest mistake in their Google career, and they said "Not using ECC memory on early servers." If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you could…

Google had done extensive studies[1]. There is roughly 3% chance of error in RAM per DIMM per year. That doesn't justify buying ECC if you have just one personal computer to worry about. However if you are in data center with 100K machines each with 8 DIMM, you are looking at about 6K machines experiencing RAM errors each day . Now if data is being replicated then these errors can propogate corrupted data in unpredic…

> There is roughly 3% chance of error in RAM per DIMM per year. That doesn't justify buying ECC if you have just one personal computer to worry about.

How do you make that leap?

Re: Should I buy ECC memory? (2015)

#73

Post by StillBored »

paulsutter wrote: post #69
lomnakkus wrote: post #29

Earlier quoted context omitted.

> If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you couldn't trust the bits that your RAM gave back to you. Details of this would be very interesting, but obviously I understand if you cannot provide such details due to NDAs, etc. I mean, I can imagine a few mitigations (pervasive checksumming,…

You need ECC /and/ pervasive checksumming. There are too many stages of processing where errors can occur. For example, disk controllers or networks. The TCP checksum is a bit of a joke at 16 bits (it will fail to detect 1 in 65000 errors), and even the Ethernet CRC can fail - you need end to end checksums. http://www.evanjones.ca/tcp-and-ethernet-checksums-fail.html

I did a bunch of protocol level design in the 90's and one of the handful of things that taught me was _ALWAYS_ use at least a CRC with a standard polynomial. Its just not worth it, in the 2000's I relearned the lesson when it comes to data at rest (on disk/etc). If nothing else both of those will catch "bugs" rather than silently corrupting things and leading to mysteries long after the initial data was corrupted.

I just had this discussion (about why TCP's checksum was a huge mistake) a couple days ago. That link is going to be useful next time it comes up.

Re: Should I buy ECC memory? (2015)

#74

Post by derefr »

lomnakkus wrote: post #29
nostrademons wrote: post #7

While I was at Google, someone asked one of the very early Googlers (I think it was Craig Silverstein, but it may've been Jeff Dean) what was the biggest mistake in their Google career, and they said "Not using ECC memory on early servers." If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you could…

> If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you couldn't trust the bits that your RAM gave back to you. Details of this would be very interesting, but obviously I understand if you cannot provide such details due to NDAs, etc. I mean, I can imagine a few mitigations (pervasive checksumming,…

> ultimately there's very little you can actually do reliably if your memory is lying to you

1. Implement everything in terms of retry-able jobs; ensure that jobs fail when they hit checksum errors.

2. if you've got a bytecode-executing VM, extend it to compare its modules to stored checksums, just before it returns from them; and to throw an exception instead of returning if it finds a problem. (This is a lot like Microsoft's stack-integrity protection, but for notionally "read-only" sections rather than read-write sections.)

3. Treat all such checksum failures as a reason to immediately halt the hardware and schedule it for RAM replacement. Ensure that your job-system handles crashed nodes by rescheduling their jobs to other nodes. If possible, also undo the completion of any recently-completed jobs that ran on that node.

4. Run regular "memtest monkey" jobs on all nodes that attempt to trigger checksum failures. To get this to work well, either:

4a. ensure that jobs die often enough, and are scheduled onto nodes in random-enough orders, that no job ever "pins" a section of physical memory indefinitely;

4b. or, alternately, write your own kernel memory-page allocation strategy, to map physical memory pages at random instead of linearly. (Your TLBs will be very full!)

Mind you, steps 3 and 4 only matter to catch persistent bit-errors (i.e. failing RAM); one-time cosmic-ray errors can only really be caught by steps 1 and 2, and even then, only if they happen to affect memory that ends up checksummed.

Re: Should I buy ECC memory? (2015)

#75

Post by thraway2016 »

throwa38282 wrote: post #62
amenod wrote: post #54

Earlier quoted context omitted.

NSA agrees with you, for one.

NSA is not the primary threat here. More conventional legal mechanisms are. Home serving is just as vulnerable to the NSA.

Conventional legal mechanisms against your home server cabinet can be handled via full disk encryption and a reed switch on your cabinet door connected to your power strip.

Re: Should I buy ECC memory? (2015)

#76

Post by jacquesm »

muro wrote: post #67
jacquesm wrote: post #48

Earlier quoted context omitted.

Maybe they are under contract not to pass information to third parties or maybe the company policy is to not let internal email off the network. That you don't understand it is likely from the perspective of an individual, possibly a private user. For those applications you can't beat the cloud. For business use every business needs to weigh their own needs. Even then though, many business think they need to have the…

Sure, but those are rare and usually include enough budget to include sysadmins and definitely enough to buy ECC memory. For anyone weighing whether ECC is worth it, they are wasting time managing their email server.

Those are not rare at all. Every lawyers office has this problem, every journalist, every banker, every insurance company, every notary public, every administration and so on.

Re: Should I buy ECC memory? (2015)

#77

Post by lucb1e »

This article is gold in so many ways. It contains interesting bits of information on ECC, company history that I didn't know (Sun's and Google's namely), filesystem reliability (I never knew!), the physics of RAM (50 electrons per capacitor)...

It's a must read, even if only to get you thinking about some of these things.

Re: Should I buy ECC memory? (2015)

#78

Post by derefr »

lomnakkus wrote: post #38
olavgg wrote: post #26

Can people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actually one of the few that can still be safer if you don't run with ECC memory. Source: Matthew Ahrens himself: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...

Indeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from. The main point of e.g. ZFS or Btrfs checksumming is that a) at least it isn't getting worse , and b) I can tell if it's getting worse. [1] ... but if the bits are not generat…

> The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.

• If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed.

• Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property.

• Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.)

• Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting them (though not the page itself) from errors.

• ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on installer ISOs embed their own checksums and often signatures.

• git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread.

Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.

Re: Should I buy ECC memory? (2015)

#79

Post by derefr »

lomnakkus wrote: post #68
ruleabidinguser wrote: post #59

Earlier quoted context omitted.

What are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.

zpool scrub (This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)

Either way, it'd be fine on SSDs, then?

Re: Should I buy ECC memory? (2015)

#80

Post by delish »

loeg wrote: post #72
sytelus wrote: post #55

Earlier quoted context omitted.

Google had done extensive studies[1]. There is roughly 3% chance of error in RAM per DIMM per year. That doesn't justify buying ECC if you have just one personal computer to worry about. However if you are in data center with 100K machines each with 8 DIMM, you are looking at about 6K machines experiencing RAM errors each day . Now if data is being replicated then these errors can propogate corrupted data in unpredic…

> There is roughly 3% chance of error in RAM per DIMM per year. That doesn't justify buying ECC if you have just one personal computer to worry about. How do you make that leap?

I'd like to know this, too.

I am guessing it's because, if RAM errors increase linearly with the number of computers, then RAM errors will be a greater and greater proportion of total errors. This assumes other kinds of errors don't scale linearly. Someone looking through logs is looking for errors, they'd like to find fixable logic errors, not inevitable RAM errors.

Post reply on HN