Redis crashes - a small rant about software reliability • Hacker News Board

Re: Redis crashes - a small rant about software reliability

#101

Post by gizzlon » Wed, Nov 28, 2012, 11:45 AM UTC

shin_lao wrote: post #98

jules wrote: post #97

Earlier quoted context omitted.

I was responding to this: "Changing the language doesn't bring anything." and this: "You cannot reduce risk, you can only replace it with another", not to the Wikipedia entry. The Wikipedia entry is very reasonable, and does not make the same outrageous claims.

"Changing the language doesn't bring anything." -> This is because the largest contributor to code quality is the coder. C is already a "safe" language as opposed to assembly or even writing the opcodes yourself. "You cannot reduce risk, you can only replace it with another" -> I just ask you to think about it, try to have an open mind about what I may imply. I'm sorry for this mysterious answer but this is typically…

"This is because the largest contributor to code quality is the coder"

Even if that's true, it doesn't support your claim.. The language still matter, even if it turns out that it's not the most important part.

Re: Redis crashes - a small rant about software reliability

#102

Post by pilgrim689 » Wed, Nov 28, 2012, 1:47 PM UTC

CrLf wrote: post #100

pilgrim689 wrote: post #10

Earlier quoted context omitted.

Sorry if I'm breaking your bubble, but Linux and git are not "the best and most robust systems in the world". If they were, the state of the art of safe and reliable software systems would be quite pitiful. edit: that doesn't detract your point however that C is used nowadays on "robust systems"... in terms of popular robust kernels though you'll want to look at something like L4 or QNX Neutrino. There's a kernel tha…

I don't even remember the last time I saw a Linux server crash... at least I can't think of any occasion in the last 10 years or so that wasn't directly related to some hardware failure (firmware bug or actual dead hardware). So, I don't understand what you mean by Linux not being safe and reliable...

I'm not saying it is not reliable, just that it is not robust relative to safety-critical software. It can be reliably used as a server without a problem, but whether or not it can be used in a manned spacecraft, for example, is another question of robustness entirely.

Re: Redis crashes - a small rant about software reliability

#103

Post by politician » Wed, Nov 28, 2012, 2:02 PM UTC

shin_lao wrote: post #94

politician wrote: post #87

Earlier quoted context omitted.

Do you have a blog? Maybe you could do a write-up.. This is the kind of geek catnip that HN likes.

We do have a blog - the subject is vast. Do you have anything in particular you would like to read about?

Benchmarks against popular alternatives, explanations of how yours avoids the mistakes of MongoDB, your principles vs the other guys, your approach re: CAP, do you use MVCC and if not why is your solution better, the workloads that your solution excels at.

Re: Redis crashes - a small rant about software reliability

#104

Post by CrLf » Wed, Nov 28, 2012, 2:26 PM UTC

pilgrim689 wrote: post #102

CrLf wrote: post #100

Earlier quoted context omitted.

I don't even remember the last time I saw a Linux server crash... at least I can't think of any occasion in the last 10 years or so that wasn't directly related to some hardware failure (firmware bug or actual dead hardware). So, I don't understand what you mean by Linux not being safe and reliable...

I'm not saying it is not reliable, just that it is not robust relative to safety-critical software. It can be reliably used as a server without a problem, but whether or not it can be used in a manned spacecraft, for example, is another question of robustness entirely.

That has little to do with robustness. Linux may very well be as robust or even more robust than any OS considered for those purposes, but the question isn't robustness, it's predictability.

I even go further and say that Linux is more robust than many of these other OSes, since it is exposed to a wider range of environments and hardware combinations of varying quality and whatever bugs they trigger. If you reduce Linux to the kind of footprint that, for instance, VxWorks or QNX have, I don't believe it would be any less reliable.

But Linux isn't predictable. Many safety-critical systems only rely on the system not crashing or misbehaving, but many others rely on real-time characteristics. Those Linux doesn't have.

Re: Redis crashes - a small rant about software reliability

#105

Post by shin_lao » Wed, Nov 28, 2012, 3:25 PM UTC

gizzlon wrote: post #101

shin_lao wrote: post #98

Earlier quoted context omitted.

"Changing the language doesn't bring anything." -> This is because the largest contributor to code quality is the coder. C is already a "safe" language as opposed to assembly or even writing the opcodes yourself. "You cannot reduce risk, you can only replace it with another" -> I just ask you to think about it, try to have an open mind about what I may imply. I'm sorry for this mysterious answer but this is typically…

" This is because the largest contributor to code quality is the coder " Even if that's true, it doesn't support your claim.. The language still matter, even if it turns out that it's not the most important part.

Then we can only agree that we disagree. I think the language matters very little, you think the opposite.

Re: Redis crashes - a small rant about software reliability

#106

Post by jules » Wed, Nov 28, 2012, 3:42 PM UTC

shin_lao wrote: post #98

jules wrote: post #97

Earlier quoted context omitted.

I was responding to this: "Changing the language doesn't bring anything." and this: "You cannot reduce risk, you can only replace it with another", not to the Wikipedia entry. The Wikipedia entry is very reasonable, and does not make the same outrageous claims.

"Changing the language doesn't bring anything." -> This is because the largest contributor to code quality is the coder. C is already a "safe" language as opposed to assembly or even writing the opcodes yourself. "You cannot reduce risk, you can only replace it with another" -> I just ask you to think about it, try to have an open mind about what I may imply. I'm sorry for this mysterious answer but this is typically…

Your first paragraph is repeating what you already said, which is exactly what I responded to. To summarize it again: that C is high level or safe compared to assembler does not mean that other languages aren't significantly higher level or safer than C.

Your second paragraph is on the border of being condescending without any actual argumentation.

Re: Redis crashes - a small rant about software reliability

#107

Post by malkia » Wed, Nov 28, 2012, 5:28 PM UTC

geal wrote: post #96

malkia wrote: post #84

Earlier quoted context omitted.

Unfortunately address space randomization techniques make this much harder.

Not necessarily. The screenshot indicates the bytes pointed by the IP, so it would still be possible to find them a binary you just built, and debug it from there.

only if this is nota relocated code, and still there can be much code duplication, especially with C++ templates/inlines

Re: Redis crashes - a small rant about software reliability

#108

Post by onetwothreefour » Wed, Nov 28, 2012, 11:52 PM UTC

dap wrote: post #43

Great post, showing admirable dedication to software reliability and a solid understanding of memory issues. One of the suggestions was that the kernel could do more. Solaris-based systems (illumos, SmartOS, OmniOS, etc.) do detect both correctable and uncorrectable memory issues. Errors may still cause a process to crash, but they also raise faults to notify system administrators what's happened. You don't have to g…

Linux kernels also do that memory thing (and have done it for years), just FYI. Either through MCE or EDAC. Not really that special.

Pro-tip: use ECC memory on servers. The end.

Re: Redis crashes - a small rant about software reliability

#109

Post by javert » Thu, Nov 29, 2012, 7:26 AM UTC

pilgrim689 wrote: post #71

javert wrote: post #54

Earlier quoted context omitted.

Because it's super widely deployed and has a very mature development process, I would expect the Linux kernel to be among the most robust software in the world. I am interested to hear what you think is more robust than Linux, setting aside seL4. Do you think QNX Neutrino is more robust? If so, why? And what else? I would expect vxWorks and other RTOSs to generally be less robust than Linux, despite typically going t…

I was going to mention safety certification[1], but if you're saying we should disregard those, then I'm not sure what you're looking for. QNX was built with real-time and safety-critical constraints in mind. Linux was not. I'm not saying QNX is generally better than linux... just that if I want to browse the web at home I'd use Linux, and if I want a kernel to control the car I drive I'd choose QNX. [1] http://www.q…

AFAIK, the kind of safety certificaitons applied to RTOSs are all about development process, and have nothing to do with actually proving correctness in a "mathematical" sense. I know very little about IEC 61508, but for DO-178B/C, it is all about development process. No certification agency wants to claim that software is actually correct, because they can't actually show that, and then they'd be blamed if something bad happened.

I have talked to people in the preempt_rt Linux community that believe Linux will come to dominate the RTOS market just like it has a lot of other stuff, and I think they have a compelling argument. Once preempt_rt is mature enough, it's hard for me to see any reason for going with something like QNX.

Re: Redis crashes - a small rant about software reliability

#110

Post by javert » Thu, Nov 29, 2012, 7:39 AM UTC

erichocean wrote: post #70

javert wrote: post #64

Earlier quoted context omitted.

I don't really understand your driver argument, because when deploying Linux on an embedded system, you would only use the drivers you need... so you'd have the same amount of driver code as with any other kernel. You say that you managed to "verify correctness" on QK, but clearly, that's not an accurate statement, since AFAIK seL4 is the only kernel that has ever been "proven correct" (and even for seL4, there are s…

The entire QK[1] kernel, including all library code, is less that two thousand lines of C (and most of the library code isn't even used by the kernel itself, which is just 234 LOC). QK is a single-processor kernel (although still pre-emptive) and very well written and tested (in, literally, hundreds of millions of devices). It was not difficult at all to run QK and it's supporting code through KLEE and exhaustively v…

Interesting. I had never heard of QK, and I'm slightly surprised about that.

people should no longer be surprised when a small-but-meaningful codebase is certified correct

For the size of codebase you're talking about, I'm not really that surprised. But I still think that formal methods don't scale to multicore RTOSs, which is the area I study (I'm a CS grad student).