Page is down. Here is a formatted copy: https://gist.github.com/4154289
The irony is too thick to read through.
Redis crashes - a small rant about software reliability
11–20 of 112 posts
Re: Redis crashes - a small rant about software reliability
#12Perhaps using safer languages (and languages with better error reporting) would be a solution to these kinds of problems.
I'm not sure why you are being downvoted, as it is still an open question whether or not safer languages make it easier to write safer programs. So your comment may be a valid one.
Re: Redis crashes - a small rant about software reliability
#13This is an interesting post, especially the part about memory testing. We have a simple policy: ECC memory is required to run our software in production. Failure to do so voids the warranty.
What if your customers want to run on EC2 instances?
Re: Redis crashes - a small rant about software reliability
#14In theory, there's nothing stopping the OS from remapping the pages of your address space to different physical RAM locations at any point during your test. So even if you have a reproducible bit error that caused the crash, there's a chance that the defect memory region is not actually touched during the memory test.
Now, this may not be such a huge problem in practice because the OS is unlikely to move pages around unless it's forced to swap. But that depends on details of the OS paging algorithm and your server load.
Re: Redis crashes - a small rant about software reliability
#15His point about logging registers and stack is interesting. Many years ago I worked on some software that ran on Windows NT 4.0 and we had a weird crash from a customer who sent in a screen shot of a GPF like this: http://pisoft.ru/verstak/insider/cwfgpf1.gif
From it I was able to figure out what was wrong with the C++ program. Notice that the GPF lists the instructions at CS:EIP (the instruction pointer of the running program) and so it was possible by generating assembler output from the C++ program to identify the function/method being executed. From the registers it was possible to identify that one of the parameters was a null pointer (something like ECX being 00000000) and from that information work back up the code to figure out under what conditions that pointer could be null.
Just from that screenshot the bug was identified and fixed.
Re: Redis crashes - a small rant about software reliability
#16This is an interesting post, especially the part about memory testing. We have a simple policy: ECC memory is required to run our software in production. Failure to do so voids the warranty.
Does Amazon or other provider offers this warranty, I mean, ECC servers (or something similar?)
Re: Redis crashes - a small rant about software reliability
#17Page is down. Here is a formatted copy: https://gist.github.com/4154289
Formatted more nicely: http://gist.io/4154289
By the way, check the page first, give bloggers the traffic their content deserves!
Re: Redis crashes - a small rant about software reliability
#18This is an interesting post, especially the part about memory testing. We have a simple policy: ECC memory is required to run our software in production. Failure to do so voids the warranty.
What if your customers want to run on EC2 instances?
It is covered in the blog post. (This is not a critique, just an hint, I understand that reading a very long blog post is time consuming).
Re: Redis crashes - a small rant about software reliability
#19Page is down. Here is a formatted copy: https://gist.github.com/4154289
Is he not caching his blog with Redis?
Re: Redis crashes - a small rant about software reliability
#20Can't agree with this more.. And he is just talking about logging crashes. One of the best debugging tools you have at your disposal in a large system (a lot of programmers contributing code -- bugs can be anywhere) is logging the same stack information in a quick fashion under normal operation in strange circumstances so as not to slow down the production software. The slowest part of printing that information out is the symbol resolution in the binary of the stack addresses to symbol names. This part of the debugging output can be done "offline" in a helper viewer binary and does not need to be done in the critical path. We frequently output stack traces as strings of hex addresses detectable by a regex appended to a log message. The log viewer transforms this back into an actual symbolic stack trace at viewing time to avoid the hit of resolving all the symbols in the hot path.