Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis crashes - a small rant about software reliability

antirez.com

Post reply on HN
Print view View original

91–100 of 112 posts

Re: Redis crashes - a small rant about software reliability

#91

Post by barrkel »

BoredAstronaut wrote: post #81

This post reminded me of my time as a consulting systems support specialist. Lots of weird problem turned out to be bad hardware. Usually memory or disk, sometimes bad logic boards. For end users, this would often lead to complete freezing of the computer, so it was less likely to be blamed on broken software, but there were still many times it was hard to be sure. Desktop OS software can flake out in strange ways du…

I've always been intrigued by the idea that the most reliable software programs are usually compilers.

I don't think this is necessarily true; I used to maintain the Delphi compiler, and there were hundreds of bugs in the backlog that never really got looked at owing to workarounds, low impact and high cost of fixing.

What compilers usually have going for them is that they are batch processes rather than online processes, so they don't have time to build up crud in data structures; they have highly reproducible inputs - code that causes a crash normally causes a crash every run of the program, no weird mouse clicks or timing needed, and this code can usually be sent back to the vendor; and all customer code is effectively a unit test, so feedback from betas etc. is immediate and loud.

Re: Redis crashes - a small rant about software reliability

#92

Post by shin_lao »

jules wrote: post #47
shin_lao wrote: post #37

Earlier quoted context omitted.

I think you're falling in to the "silver bullet" trap. http://en.wikipedia.org/wiki/No_Silver_Bullet Basically, making reliable software is hard. Changing the language doesn't bring anything. There are a lot of tools to make sure your C/C++ programs doesn't have obvious errors. The problem are non-obvious errors, and these errors exist in all the languages, with different forms. Another way to put it: "You cannot red…

This is obviously not true. Language matters. Are you as productive in BrainFuck as in C++? I don't think so. C/C++ isn't the end of all progress in language design. Modern languages eliminate or vastly reduce whole classes of concerns like memory management, wild pointers, null pointers, incomprehensible template errors, etc. Some of these problems can be patched up for perhaps 30% by a post hoc static analysis, but…

Please read the wikipedia entry which addresses your specific concern regarding programming languages.

Re: Redis crashes - a small rant about software reliability

#93

Post by tylerneylon »

The memory check algorithm is a nice solution of the challenges he presents - easy to understand and effective.

Here is a variation which, unless I'm missing something, would be a little simpler still and require less full-memory loops:

1. Count #1's in memory (possibly mod N to avoid overflow). 2. Invert memory. 3. Count #0's in memory. 4 Invert memory.

I think this would catch the same errors (stuck-as-0 or stuck-as-1 bits).

One difficulty is that multiple errors could cancel each other out, at which point you can do things like add checkpoints in the aggregation, or track more signals such as number of 01's vs number of 10's. In the end, this is like an inversion-friendly CRC.

Re: Redis crashes - a small rant about software reliability

#94

Post by shin_lao »

politician wrote: post #87
shin_lao wrote: post #68

Earlier quoted context omitted.

Thanks for the feedback, this is currently a landing page we give to our customers we meet face to face. We're working on something more consistent to answer questions like yours. quasardb is a key/value store. It is (a lot) faster in a multi-client context that the engines you listed and can handle entries of any size (provided you have enough space on the servers, of course!). It's fully symmetric which means the l…

Do you have a blog? Maybe you could do a write-up.. This is the kind of geek catnip that HN likes.

We do have a blog - the subject is vast. Do you have anything in particular you would like to read about?

Re: Redis crashes - a small rant about software reliability

#95

Post by [deleted] »

politician wrote: post #86
sliverstorm wrote: post #82

Earlier quoted context omitted.

I, for one, replace memory modules as soon as they trigger more than one ECC event. I thought ECC events were triggered by environment, rather than hardware faults? Or you just figure some sticks are by chance more susceptible?

Manufacturing memory from silicon wafers is similar to baking cookies. Some cookies are great, some turn out OK, and some are burnt depending on the characteristics of the ingredients, the oven, and the chaotic thermodynamic properties of the system. So, yes, yield varies.

[deleted]

Re: Redis crashes - a small rant about software reliability

#96

Post by geal »

malkia wrote: post #84
jgrahamc wrote: post #15

His point about logging registers and stack is interesting. Many years ago I worked on some software that ran on Windows NT 4.0 and we had a weird crash from a customer who sent in a screen shot of a GPF like this: http://pisoft.ru/verstak/insider/cwfgpf1.gif From it I was able to figure out what was wrong with the C++ program. Notice that the GPF lists the instructions at CS:EIP (the instruction pointer of the runni…

Unfortunately address space randomization techniques make this much harder.

Not necessarily. The screenshot indicates the bytes pointed by the IP, so it would still be possible to find them a binary you just built, and debug it from there.

Re: Redis crashes - a small rant about software reliability

#97

Post by jules »

shin_lao wrote: post #92
jules wrote: post #47

Earlier quoted context omitted.

This is obviously not true. Language matters. Are you as productive in BrainFuck as in C++? I don't think so. C/C++ isn't the end of all progress in language design. Modern languages eliminate or vastly reduce whole classes of concerns like memory management, wild pointers, null pointers, incomprehensible template errors, etc. Some of these problems can be patched up for perhaps 30% by a post hoc static analysis, but…

Please read the wikipedia entry which addresses your specific concern regarding programming languages.

I was responding to this: "Changing the language doesn't bring anything." and this: "You cannot reduce risk, you can only replace it with another", not to the Wikipedia entry. The Wikipedia entry is very reasonable, and does not make the same outrageous claims.

Re: Redis crashes - a small rant about software reliability

#98

Post by shin_lao »

jules wrote: post #97
shin_lao wrote: post #92

Earlier quoted context omitted.

Please read the wikipedia entry which addresses your specific concern regarding programming languages.

I was responding to this: "Changing the language doesn't bring anything." and this: "You cannot reduce risk, you can only replace it with another", not to the Wikipedia entry. The Wikipedia entry is very reasonable, and does not make the same outrageous claims.

"Changing the language doesn't bring anything." -> This is because the largest contributor to code quality is the coder. C is already a "safe" language as opposed to assembly or even writing the opcodes yourself.

"You cannot reduce risk, you can only replace it with another" -> I just ask you to think about it, try to have an open mind about what I may imply. I'm sorry for this mysterious answer but this is typically a topic over which we could talk past each other. The first time I was told you I had the same reaction until I had the epiphany about what it really means.

Re: Redis crashes - a small rant about software reliability

#99

Post by CrLf »

sliverstorm wrote: post #82
CrLf wrote: post #78

Earlier quoted context omitted.

You are factually correct. However, availability isn't the problem ECC memory intends to solve. The problem with memory errors is that they are silent. You won't notice them until something goes misteriously wrong. And that can be anything, from the innocent invalid memory access to data corruption. This just can't be tolerated anywhere data is being processed, data you don't want to lose that is... RAID does nothing…

I, for one, replace memory modules as soon as they trigger more than one ECC event. I thought ECC events were triggered by environment, rather than hardware faults? Or you just figure some sticks are by chance more susceptible?

ECC events are triggered by any memory error, be it the occasional cosmic ray or a not so good memory module.

It isn't difficult to tell these two possibilities apart. Sometimes I get an ECC event on some server, and then it never happens again (or it happens in a different module), which doesn't warrant a replacement. Now, if the same module triggers another event, what's the chance of two "cosmic rays" hitting the same module twice and flipping a bit on it? It's better to just replace it (which is covered by warranty or maintenance contracts, so it costs us no additional charge).

Re: Redis crashes - a small rant about software reliability

#100

Post by CrLf »

pilgrim689 wrote: post #10
wheaties wrote: post #7

Earlier quoted context omitted.

Um, no. C is a perfectly valid language and some of the best, most robust systems in the world are written in it (Linux, Git, etc.) Some languages are even built to run atop C (Cython.) Even the JVM deals with pointers, memory allocation issues, and such so you don't have to but it's still there! So using a higher level or "safer" language isn't going to stop these kinds of problems.

Sorry if I'm breaking your bubble, but Linux and git are not "the best and most robust systems in the world". If they were, the state of the art of safe and reliable software systems would be quite pitiful. edit: that doesn't detract your point however that C is used nowadays on "robust systems"... in terms of popular robust kernels though you'll want to look at something like L4 or QNX Neutrino. There's a kernel tha…

I don't even remember the last time I saw a Linux server crash... at least I can't think of any occasion in the last 10 years or so that wasn't directly related to some hardware failure (firmware bug or actual dead hardware).

So, I don't understand what you mean by Linux not being safe and reliable...

Post reply on HN