Live data from Hacker News

Redis crashes - a small rant about software reliability

antirez.com

71–80 of 112 posts

Re: Redis crashes - a small rant about software reliability

#71
post #54

Earlier quoted context omitted.

Sorry if I'm breaking your bubble, but Linux and git are not "the best and most robust systems in the world". If they were, the state of the art of safe and reliable software systems would be quite pitiful. edit: that doesn't detract your point however that C is used nowadays on "robust systems"... in terms of popular robust kernels though you'll want to look at something like L4 or QNX Neutrino. There's a kernel tha…

Because it's super widely deployed and has a very mature development process, I would expect the Linux kernel to be among the most robust software in the world. I am interested to hear what you think is more robust than Linux, setting aside seL4. Do you think QNX Neutrino is more robust? If so, why? And what else? I would expect vxWorks and other RTOSs to generally be less robust than Linux, despite typically going t…

I was going to mention safety certification[1], but if you're saying we should disregard those, then I'm not sure what you're looking for. QNX was built with real-time and safety-critical constraints in mind. Linux was not. I'm not saying QNX is generally better than linux... just that if I want to browse the web at home I'd use Linux, and if I want a kernel to control the car I drive I'd choose QNX.

[1] http://www.qnx.com/products/neutrino-rtos/safe-kernel.html

Re: Redis crashes - a small rant about software reliability

#72
post #69

I find this idea of a lack of ECC memory on servers disturbing... This is the default on almost all rack mountable servers from the likes of HP or IBM. Of course, people use all kinds of sub-standard hardware for "servers" on the cheap, and they get what they pay for. I haven't seen a server without ECC memory for years. I don't even consider running anything in production without ECC memory, let alone VM hypervisors…

I hope there is a way to get some official statement from Amazon, Linode, and other very used VM providers about the kind of memory used in their servers. This would help users understanding the real risks.

Re: Redis crashes - a small rant about software reliability

#73
post #43

Great post, showing admirable dedication to software reliability and a solid understanding of memory issues. One of the suggestions was that the kernel could do more. Solaris-based systems (illumos, SmartOS, OmniOS, etc.) do detect both correctable and uncorrectable memory issues. Errors may still cause a process to crash, but they also raise faults to notify system administrators what's happened. You don't have to g…

I don't think enough people appreciate just how awesome of an OS Solaris was. I never had opportunity to deploy it full-scale for any projects, but I lamented the loss of great potential when it "died."

Re: Redis crashes - a small rant about software reliability

#74
post #54

Earlier quoted context omitted.

Because it's super widely deployed and has a very mature development process, I would expect the Linux kernel to be among the most robust software in the world. I am interested to hear what you think is more robust than Linux, setting aside seL4. Do you think QNX Neutrino is more robust? If so, why? And what else? I would expect vxWorks and other RTOSs to generally be less robust than Linux, despite typically going t…

The major source of errors in a kernel is device drivers, and Linux typically is running many more drivers than the embedded kernels you mentioned. Look at any Linux point release: the majority of churn is in driver code, to fix bugs. Thus, it stands to reason, Linux is likely less stable than an embedded kernel without all that driver code. I run a pre-emptive embedded kernel (QK) that's extremely tiny and, in fact,…

So I think it'd be fair to say that Linux does ok in the robustness / lines of code or robustness / devices and real world interfaces, however, in terms of absolute numbers, a smaller system is almost always going to do better.

Re: Redis crashes - a small rant about software reliability

#75
post #69

I find this idea of a lack of ECC memory on servers disturbing... This is the default on almost all rack mountable servers from the likes of HP or IBM. Of course, people use all kinds of sub-standard hardware for "servers" on the cheap, and they get what they pay for. I haven't seen a server without ECC memory for years. I don't even consider running anything in production without ECC memory, let alone VM hypervisors…

A single machine is never going to be completely reliable. At any time it can halt for a variety of reasons: power loss, hardware failure, disaster in the data center like flooding, etc.

Thus, a configuration that relies on the availability of a single machine is already risking serious outage or data loss by not being machine-redundant. Reliable systems require the coordination of many machines (at least two), and the replication of data across them if data's involved.

It is useful to have component-level redundancy (e.g., RAID or ECC memory), but in some environments it may be cheaper overall to have machine-level redundancy using inexpensive machines. It also only takes the failure of a single critical subsystem for a machine to suffer an outage. You might have ECC memory and RAID, but do you have only a single Ethernet card and power supply? Single machine availability is a "weakest link" phenomenon from its components.

I acknowledge that building software to run across a fleet of machines is more difficult than software that runs on only a single machine, but (1) the software development cost is largely a fixed cost, not a variable cost in the number of machines (2) building a distributed system is sometimes needed for scaling reasons anyway.

If you scale a single machine vertically (i.e., get a bigger box), its cost rises faster than its capabilities; so an efficient high-scale system typically also means running a fleet of cheap machines (scale horizontally). I think these effects contribute to the rise of commodity-server computing, and cost is a reason not to consider it disturbing.

In other words, crunch the numbers and see when it makes sense :-)

Re: Redis crashes - a small rant about software reliability

#76
post #69

I find this idea of a lack of ECC memory on servers disturbing... This is the default on almost all rack mountable servers from the likes of HP or IBM. Of course, people use all kinds of sub-standard hardware for "servers" on the cheap, and they get what they pay for. I haven't seen a server without ECC memory for years. I don't even consider running anything in production without ECC memory, let alone VM hypervisors…

[deleted]

Re: Redis crashes - a small rant about software reliability

#77
post #43

Great post, showing admirable dedication to software reliability and a solid understanding of memory issues. One of the suggestions was that the kernel could do more. Solaris-based systems (illumos, SmartOS, OmniOS, etc.) do detect both correctable and uncorrectable memory issues. Errors may still cause a process to crash, but they also raise faults to notify system administrators what's happened. You don't have to g…

I don't think enough people appreciate just how awesome of an OS Solaris was. I never had opportunity to deploy it full-scale for any projects, but I lamented the loss of great potential when it "died."

It didn't die. It was forked by the community when Oracle close-sourced it. The community fork (called illumos) is being actively developed by multiple companies, which have done significant new feature work (e.g., http://dtrace.org/blogs/wdp/2011/03/our-zfs-io-throttle/).

Re: Redis crashes - a small rant about software reliability

#78
post #75
post #69

I find this idea of a lack of ECC memory on servers disturbing... This is the default on almost all rack mountable servers from the likes of HP or IBM. Of course, people use all kinds of sub-standard hardware for "servers" on the cheap, and they get what they pay for. I haven't seen a server without ECC memory for years. I don't even consider running anything in production without ECC memory, let alone VM hypervisors…

A single machine is never going to be completely reliable. At any time it can halt for a variety of reasons: power loss, hardware failure, disaster in the data center like flooding, etc. Thus, a configuration that relies on the availability of a single machine is already risking serious outage or data loss by not being machine-redundant. Reliable systems require the coordination of many machines (at least two), and t…

You are factually correct. However, availability isn't the problem ECC memory intends to solve.

The problem with memory errors is that they are silent. You won't notice them until something goes misteriously wrong. And that can be anything, from the innocent invalid memory access to data corruption. This just can't be tolerated anywhere data is being processed, data you don't want to lose that is...

RAID does nothing if the OS thinks that its in-memory filesystem datastructures are correct, and just goes ahead and updates the superblock with bad data, or writes over other files' pages. You just get a nice, redundant, corrupted filesystem. The same goes for multiple machines sharing data anywhere, filesystems or databases alike.

It's the error detection part that's important, not the correction part. And ECC main memory is just a part of the picture, you want to be notified of errors as soon as possible. And this is the important bit: "be notified". So, you want parity checks and CRCs on disk caches and data buses and everywhere else it's feasible. It's not an accident that server-class hardware costs more than your average PC.

The "correction" part is just a welcome by-product. I, for one, replace memory modules as soon as they trigger more than one ECC event. And this happens occasionally even with an universe of machines in the low dozens, with supposedly high-quality components. Now think what may be happening silently with all those borderline memory modules from anonymous manufacturers in China...

Besides, like I mentioned before, it isn't easy to find non-ECC memory servers from the usual vendors. Only their very low-end machines have it. Machines that aren't meant to do anything more that shoving packets around or other usage patterns where either silent data corruption can be tolerated (easy to replace appliances that don't process/store important data) or checksums are already a part of the job (network stuff like firewalls or routers).

Re: Redis crashes - a small rant about software reliability

#79
post #72
post #69

I find this idea of a lack of ECC memory on servers disturbing... This is the default on almost all rack mountable servers from the likes of HP or IBM. Of course, people use all kinds of sub-standard hardware for "servers" on the cheap, and they get what they pay for. I haven't seen a server without ECC memory for years. I don't even consider running anything in production without ECC memory, let alone VM hypervisors…

I hope there is a way to get some official statement from Amazon, Linode, and other very used VM providers about the kind of memory used in their servers. This would help users understanding the real risks.

I think they don't mention it because they think it to be obvious (I hope). However, with all the special built servers that big providers use to reduce costs, there is some margin to doubt.

I think Google may be able to get away with it. With enough checksums along the way, memory (and other hardware) errors can be detected in software pretty easily if you have independent machines checking the data and can afford the processing penalty.

Now, for virtualization I seriously doubt it. Not unless their instances run simultaneously on more than one machine to check for inconsistencies between them (something that the mainframes do since the dawn of time, but that I don't see as feasible in a distributed environment).

Re: Redis crashes - a small rant about software reliability

#80
And people wonder why I recommend redis. Having run redis for over 1.5 years on production systems as a heavy cache, a named queue and memoization tool (on the same machine), redis has never once failed me. It's clear with antirez's blog post, his attention to detail.

This post is fantastic.

Post reply on HN