Live data from Hacker News

Redis crashes - a small rant about software reliability

antirez.com

31–40 of 112 posts

Re: Redis crashes - a small rant about software reliability

#31
post #18
post #13

Earlier quoted context omitted.

What if your customers want to run on EC2 instances?

It is covered in the blog post. (This is not a critique, just an hint, I understand that reading a very long blog post is time consuming).

I did read the whole post. It was very informative. I wasn't aware that EC2 did not have ECC RAM. My question was directed at shin_lao's policy about not providing a "warranty" for his customers running on non-ECC hardware.

Re: Redis crashes - a small rant about software reliability

#32

Perhaps using safer languages (and languages with better error reporting) would be a solution to these kinds of problems.

Such languages also allow you to write code much faster which can lead to more mistakes. Edit: huh. 2 downvotes. Why not explain why you think I'm wrong, rather than just downvoting because you don't agree?

You get downvotes because your reply contradicts itself. A safer language results in more mistakes being found at compilation time, so would result in less mistakes in the product, not more. Also I would expect having to specify annotations for your code to be verified (ie correct type specifications, pre/postconditions, contracts, carefully specifying program input/output through parsers, etc) results in slightly slower pace of development, not faster. But I suppose if you make up more than that time in debugging and troubleshooting time, or time explaing awkward bugs to annoyed customers, it's still a win.

Re: Redis crashes - a small rant about software reliability

#33
post #31
post #18

Earlier quoted context omitted.

It is covered in the blog post. (This is not a critique, just an hint, I understand that reading a very long blog post is time consuming).

I did read the whole post. It was very informative. I wasn't aware that EC2 did not have ECC RAM. My question was directed at shin_lao's policy about not providing a "warranty" for his customers running on non-ECC hardware.

Oh sorry I get it now...

Re: Redis crashes - a small rant about software reliability

#34

Perhaps using safer languages (and languages with better error reporting) would be a solution to these kinds of problems.

Much of the article is dedicated to hardware level memory problems. Most of the 'safer' languages I know about would not really help you in those situations. Maybe there are some odd mainframe languages that read and write everything twice or something like that, but most 'safer' languages I know about are more oriented towards preventing programmer errors. What antirez is saying that, yes, they get some of those, but many others, due to how widely used the system is, that are genuine memory failures, that cost a lot of time to track down.

Re: Redis crashes - a small rant about software reliability

#35
I totally like this post, because main-memory based software systems will become the future for all kinds of applications. Thus, handling errors on this side will become more important as well.

Here are my additional two cents: At least on X86 systems, to check small memory regions without effects on the CPU cache can be implemented using non-temporal writes that will directly force the CPU to write the memory back to memory. The instruction required for this is called movntdq and is generated by the SSE2 intrinsic _mm_stream_si128().

Re: Redis crashes - a small rant about software reliability

#36

Perhaps using safer languages (and languages with better error reporting) would be a solution to these kinds of problems.

Well, not to the memory problems. But I do think modern systems languages should be better about printing out stuff in the case of segfaults, for example. Not sure if Go and Rust do this stuff.

Re: Redis crashes - a small rant about software reliability

#37

Earlier quoted context omitted.

Rust isn't production ready.

I never claimed it was. I'm just thinking that perhaps the approach it (and Haskell, for that matter) takes is better.

I think you're falling in to the "silver bullet" trap.

http://en.wikipedia.org/wiki/No_Silver_Bullet

Basically, making reliable software is hard. Changing the language doesn't bring anything. There are a lot of tools to make sure your C/C++ programs doesn't have obvious errors. The problem are non-obvious errors, and these errors exist in all the languages, with different forms.

Another way to put it: "You cannot reduce risk, you can only replace it with another".

Re: Redis crashes - a small rant about software reliability

#38

Earlier quoted context omitted.

I never claimed it was. I'm just thinking that perhaps the approach it (and Haskell, for that matter) takes is better.

I think you're falling in to the "silver bullet" trap. http://en.wikipedia.org/wiki/No_Silver_Bullet Basically, making reliable software is hard. Changing the language doesn't bring anything. There are a lot of tools to make sure your C/C++ programs doesn't have obvious errors. The problem are non-obvious errors, and these errors exist in all the languages, with different forms. Another way to put it: "You cannot red…

I agree that nothing solves all possible problems. But it's a huge jump from "there is no silver bullet" to "changing the language doesn't bring ANYTHING". Likely, it is quite a bit more subtle in practice. Preventing "obvious" errors (which really aren't always that obvious) that safe languages could minimize is already a great start, as many systems are rife with them.

Re: Redis crashes - a small rant about software reliability

#39
At IBM, we were very keen on what we called 'FFDC' - 'first- failure data capture'. This meant having enough layers of error-detection, ideally all the way down to the metal, so that failures could be detected cleanly and logged before (possibly) going down, allowing our devs to reproduce and fix customer bugs. Naturally it wasn't perfect, and it depending on lots of very tedious planning meetings, but on the stuff I worked with (storage devices mainly) it was remarkably effective.

In my experience in more 'agile' firms - startups, web dev shops and so on - it would be very hard to make a scheme like this work well, because of all the grinding bureaucracy, fiddly spec-matching and endless manual testing required, as well as the importance of controlling - and deeply understanding - the whole stack. Nonetheless, for infrastructure projects like Redis, I can see value in having engineering effort put explicitly into making 'prettier crashes'.

Re: Redis crashes - a small rant about software reliability

#40

His point about logging registers and stack is interesting. Many years ago I worked on some software that ran on Windows NT 4.0 and we had a weird crash from a customer who sent in a screen shot of a GPF like this: http://pisoft.ru/verstak/insider/cwfgpf1.gif From it I was able to figure out what was wrong with the C++ program. Notice that the GPF lists the instructions at CS:EIP (the instruction pointer of the runni…

I remember generating .map files as part of the build process that were invaluable in figuring out where Windows desktop programs were crashing. It was about 30 minutes of work that made 3-hour debugging sessions into 10 minute debugging sessions from then on.
Post reply on HN