Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis crashes - a small rant about software reliability

antirez.com

Post reply on HN
Print view View original

41–50 of 112 posts

Re: Redis crashes - a small rant about software reliability

#41

Post by dap »

lucian1900 wrote: post #22
meaty wrote: post #4

Earlier quoted context omitted.

Not really. We get all the same sorts of errors in our very high level C#/Asp.Net/VMware deployments and it's a shit load harder to debug with all the extra baggage that a VM and hypervisor throw on top as well... A better solution to all the reliability problems is better quality hardware i.e. not X86. X86 has very few reliability features built in past ECC. If you look at UltraSparc based machines, they can predict…

I was thinking more of something like Rust, where only a small subset of your code would be poking at memory manually. Then when you do get a mysterious crash, you only need to look at the unsafe portions of your application (or perhaps at the compiler).

Maybe this was just the bias of the post, but it sounded from the post that most time is spent ruling out actual memory errors. Programming in a higher-level language doesn't make such errors less likely or easier to identify.

It's true that there are certain classes of errors that "safe" languages make less likely or impossible. I'm not convinced there are enough fewer of these to make up for the additional classes of errors introduced by such languages.

Re: Redis crashes - a small rant about software reliability

#42

Post by erichocean »

Although we use ECC in our servers already, I've recently been experimenting with hashing object contents in memory using a CityHash variant. The hash is checked when the object moves on chip (into cache), and re-computed before the object is stored back into RAM when it's been updated.

Although our production code is written in C, I'm not particularly worried about detecting wild writes, because we use pointer checking algorithms to detect/prevent them in the compiler. (Of course, that could be buggy too...)

What I'm trying to catch are wild writes from other devices that have access to RAM. Anyway, this is far from production code so far, but hashing has already been very successful at keeping data structures on disk consistent (a la ZFS, git), so applying the same approach to memory seems like the next step.

The speed hit is surprisingly low, 10-20%, and when you put it that way, it's like running your software on a 6 month old computer. So much of the safety stuff we refuse to do "for performance" would be like running on top-of-the-line hardware three years ago, but safely. That seems like a worthwhile trade to me...

P.s. Are people really not burning in their server hardware with memtest86? We run it for 7 days on all new hardware, and I figured that was pretty standard...

Re: Redis crashes - a small rant about software reliability

#43

Post by dap »

Great post, showing admirable dedication to software reliability and a solid understanding of memory issues.

One of the suggestions was that the kernel could do more. Solaris-based systems (illumos, SmartOS, OmniOS, etc.) do detect both correctable and uncorrectable memory issues. Errors may still cause a process to crash, but they also raise faults to notify system administrators what's happened. You don't have to guess whether you experienced a DIMM failure. After such errors, the OS then removes faulty pages from service. Of course, none of this has any performance impact until an error occurs, and then the impact is pretty minimal.

There's a fuller explanation here: https://blogs.oracle.com/relling/entry/analysis_of_memory_pa...

Re: Redis crashes - a small rant about software reliability

#44

Post by nicpottier »

This kind of attention to detail is all too rare these days. I love Redis, because I have never, not once, ever had to wonder whether it was doing its job. It is like a constant, always running, always doing a good job and getting out of the way.

It only does a few things, but it does them exceedingly well. Just like nginx, I know it will be fast and reliable, and it is this kind of crazed attention to detail that gets it there.

Re: Redis crashes - a small rant about software reliability

#45

Post by antirez »

dap wrote: post #43

Great post, showing admirable dedication to software reliability and a solid understanding of memory issues. One of the suggestions was that the kernel could do more. Solaris-based systems (illumos, SmartOS, OmniOS, etc.) do detect both correctable and uncorrectable memory issues. Errors may still cause a process to crash, but they also raise faults to notify system administrators what's happened. You don't have to g…

Thank you for the interesting link dap.

Re: Redis crashes - a small rant about software reliability

#46

Post by akx »

antirez wrote: post #27
ComputerGuru wrote: post #9

Page is down. Here is a formatted copy: https://gist.github.com/4154289

Sorry, the Sinatra based site is deployed with "ruby app.rb". Probably not enough...

May I recommend uWSGI for hosting?

http://uwsgi-docs.readthedocs.org/en/latest/Ruby.html#runnin...

Re: Redis crashes - a small rant about software reliability

#47

Post by jules »

shin_lao wrote: post #37
lucian1900 wrote: post #29

Earlier quoted context omitted.

I never claimed it was. I'm just thinking that perhaps the approach it (and Haskell, for that matter) takes is better.

I think you're falling in to the "silver bullet" trap. http://en.wikipedia.org/wiki/No_Silver_Bullet Basically, making reliable software is hard. Changing the language doesn't bring anything. There are a lot of tools to make sure your C/C++ programs doesn't have obvious errors. The problem are non-obvious errors, and these errors exist in all the languages, with different forms. Another way to put it: "You cannot red…

This is obviously not true. Language matters. Are you as productive in BrainFuck as in C++? I don't think so. C/C++ isn't the end of all progress in language design. Modern languages eliminate or vastly reduce whole classes of concerns like memory management, wild pointers, null pointers, incomprehensible template errors, etc. Some of these problems can be patched up for perhaps 30% by a post hoc static analysis, but the effectiveness of that is much less than a language that has been designed to make these errors impossible by construction. Modern languages also make lots of stuff easier through a good standard library, language features like closures & type inference, a REPL, a good IDE (the feasibility of which is also a function of language design), etc.

Sure, they don't solve the problem of magically and instantly conjuring up the program you want; nobody is claiming that. All these little things add up nevertheless.

Re: Redis crashes - a small rant about software reliability

#49

Post by marshray »

lucian1900 wrote: post #3

Perhaps using safer languages (and languages with better error reporting) would be a solution to these kinds of problems.

I think what you're missing is that antirez has gotten his C codebase perfect enough that hardware failures are the cause of most of his crash reports.

See also: http://blogs.msdn.com/b/oldnewthing/archive/2005/04/12/40756...

Re: Redis crashes - a small rant about software reliability

#50

Post by aidenn0 »

erichocean wrote: post #42

Although we use ECC in our servers already, I've recently been experimenting with hashing object contents in memory using a CityHash variant. The hash is checked when the object moves on chip (into cache), and re-computed before the object is stored back into RAM when it's been updated. Although our production code is written in C, I'm not particularly worried about detecting wild writes, because we use pointer check…

1) Yes, lots of people don't run memtest86 at all.

2) Even those that do run it typically run it for no more than 24 hours

3) Many people don't build their own hardware these days, its a VPS or EC2

4) If you've selected ECC RAM then you know way more about memory failures than >99% of Redis users

Post reply on HN