Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis crashes - a small rant about software reliability

antirez.com

Post reply on HN
Print view View original

51–60 of 112 posts

Re: Redis crashes - a small rant about software reliability

#51

Post by javert »

shin_lao wrote: post #37
lucian1900 wrote: post #29

Earlier quoted context omitted.

I never claimed it was. I'm just thinking that perhaps the approach it (and Haskell, for that matter) takes is better.

I think you're falling in to the "silver bullet" trap. http://en.wikipedia.org/wiki/No_Silver_Bullet Basically, making reliable software is hard. Changing the language doesn't bring anything. There are a lot of tools to make sure your C/C++ programs doesn't have obvious errors. The problem are non-obvious errors, and these errors exist in all the languages, with different forms. Another way to put it: "You cannot red…

> You cannot reduce risk, you can only replace it with another".

I don't understand why you would say this. Is it not the case that using a language with automatic memory management (say, Python) is less risky than using one with manual memory manamgement (say, C or C++)?

There are tradeoffs (e.g. performance, having less control over various things), but they are not tradeoffs between one risk and another risk, they are tradeoffs between risk and something else.

Re: Redis crashes - a small rant about software reliability

#52

Post by jacquesm »

antirez wrote: post #45
dap wrote: post #43

Great post, showing admirable dedication to software reliability and a solid understanding of memory issues. One of the suggestions was that the kernel could do more. Solaris-based systems (illumos, SmartOS, OmniOS, etc.) do detect both correctable and uncorrectable memory issues. Errors may still cause a process to crash, but they also raise faults to notify system administrators what's happened. You don't have to g…

Thank you for the interesting link dap.

I take it you know about /var/log/mcelog ?

Re: Redis crashes - a small rant about software reliability

#53

Post by henrikschroder »

shin_lao wrote: post #23
minimax wrote: post #13

Earlier quoted context omitted.

What if your customers want to run on EC2 instances?

On production servers using virtual machines to run our software is not advised. Nevertheless, we would do our best to please a customer looking to host our software on an EC2 cluster, with the appropriate warnings. ;) A bit of context: we sell a "real time" non-relational database ( http://www.quasardb.net/ ). Our customers come to us for speed and reliability and therefore build dedicated farms to host our database…

Wow, that product page is completely lacking any meaningful technical information about your product. :-D

How do you stack up against the most common open source NoSQL systems? Redis, Cassandra, Mongo, Couchbase? Is your db eventually consistent, or partitioned, or replicated, or what?

Re: Redis crashes - a small rant about software reliability

#54

Post by javert »

pilgrim689 wrote: post #10
wheaties wrote: post #7

Earlier quoted context omitted.

Um, no. C is a perfectly valid language and some of the best, most robust systems in the world are written in it (Linux, Git, etc.) Some languages are even built to run atop C (Cython.) Even the JVM deals with pointers, memory allocation issues, and such so you don't have to but it's still there! So using a higher level or "safer" language isn't going to stop these kinds of problems.

Sorry if I'm breaking your bubble, but Linux and git are not "the best and most robust systems in the world". If they were, the state of the art of safe and reliable software systems would be quite pitiful. edit: that doesn't detract your point however that C is used nowadays on "robust systems"... in terms of popular robust kernels though you'll want to look at something like L4 or QNX Neutrino. There's a kernel tha…

Because it's super widely deployed and has a very mature development process, I would expect the Linux kernel to be among the most robust software in the world.

I am interested to hear what you think is more robust than Linux, setting aside seL4. Do you think QNX Neutrino is more robust? If so, why? And what else?

I would expect vxWorks and other RTOSs to generally be less robust than Linux, despite typically going through various certifications.

Re: Redis crashes - a small rant about software reliability

#55

Post by javert »

jt2190 wrote: post #28
lucian1900 wrote: post #3

Perhaps using safer languages (and languages with better error reporting) would be a solution to these kinds of problems.

(Forgive me if I've got some technical details wrong and I'm not a C programmer, but I do feel like this is a valid question and isn't trying to tweak the noses of C programmers. C programmers please feel free to add/correct anything I'm missing.) C has a powerful/unsafe feature in that it allows you to directly address memory and read and write data. From a high-level, there are potential problems that can happen wh…

* I do feel like this is a valid question*

What is your actual question?

Re: Redis crashes - a small rant about software reliability

#57

Post by pcwalton »

dap wrote: post #41
lucian1900 wrote: post #22

Earlier quoted context omitted.

I was thinking more of something like Rust, where only a small subset of your code would be poking at memory manually. Then when you do get a mysterious crash, you only need to look at the unsafe portions of your application (or perhaps at the compiler).

Maybe this was just the bias of the post, but it sounded from the post that most time is spent ruling out actual memory errors. Programming in a higher-level language doesn't make such errors less likely or easier to identify. It's true that there are certain classes of errors that "safe" languages make less likely or impossible. I'm not convinced there are enough fewer of these to make up for the additional classes…

What additional classes of errors does e.g. Rust introduce?

Re: Redis crashes - a small rant about software reliability

#58

Post by kaolinite »

wladimir wrote: post #32
kaolinite wrote: post #5

Earlier quoted context omitted.

Such languages also allow you to write code much faster which can lead to more mistakes. Edit: huh. 2 downvotes. Why not explain why you think I'm wrong, rather than just downvoting because you don't agree?

You get downvotes because your reply contradicts itself. A safer language results in more mistakes being found at compilation time, so would result in less mistakes in the product, not more. Also I would expect having to specify annotations for your code to be verified (ie correct type specifications, pre/postconditions, contracts, carefully specifying program input/output through parsers, etc) results in slightly sl…

Compilers can't detect logic errors which tend to be more common with faster development.

Re: Redis crashes - a small rant about software reliability

#59

Post by erichocean »

javert wrote: post #54
pilgrim689 wrote: post #10

Earlier quoted context omitted.

Sorry if I'm breaking your bubble, but Linux and git are not "the best and most robust systems in the world". If they were, the state of the art of safe and reliable software systems would be quite pitiful. edit: that doesn't detract your point however that C is used nowadays on "robust systems"... in terms of popular robust kernels though you'll want to look at something like L4 or QNX Neutrino. There's a kernel tha…

Because it's super widely deployed and has a very mature development process, I would expect the Linux kernel to be among the most robust software in the world. I am interested to hear what you think is more robust than Linux, setting aside seL4. Do you think QNX Neutrino is more robust? If so, why? And what else? I would expect vxWorks and other RTOSs to generally be less robust than Linux, despite typically going t…

The major source of errors in a kernel is device drivers, and Linux typically is running many more drivers than the embedded kernels you mentioned. Look at any Linux point release: the majority of churn is in driver code, to fix bugs.

Thus, it stands to reason, Linux is likely less stable than an embedded kernel without all that driver code.

I run a pre-emptive embedded kernel (QK) that's extremely tiny and, in fact, was validated by myself with KLEE (a symbolic checker) to exhaustively verify correctness. I'm certain it's more reliable than Linux, which carries no such guarantee (and is orders of magnitude larger -- even excluding driver code).

Bug rates correlate extremely close with lines of code. All else being equal, a large system has more bugs, simply because it has more opportunity for them.

If you truly care about correctness, doing formal verification, model checking, etc. is the way to go, not "lots of people use it so it must be stable".

To benefit from formal verification, you have to design your code around that, and most systems today are not. It's hard to retrofit verification on top of a legacy codebase like Linux.

Re: Redis crashes - a small rant about software reliability

#60

Post by ricardobeat »

jimwhitson wrote: post #39

At IBM, we were very keen on what we called 'FFDC' - 'first- failure data capture'. This meant having enough layers of error-detection, ideally all the way down to the metal, so that failures could be detected cleanly and logged before (possibly) going down, allowing our devs to reproduce and fix customer bugs. Naturally it wasn't perfect, and it depending on lots of very tedious planning meetings, but on the stuff I…

spec-matching is a specialty of good agile companies, but web-dev shops don't usually write their own db/server software.
Post reply on HN