31–40 of 40 posts
The opening about availability and our expectations hits right at home. I'm originally from France, where stores closing during lunch time, sundays and mondays all day is normal. I remember when I moved to Canada, it seemed so amazing that stores were opened 7/7, and sometimes until 9pm (on Thursdays and Fridays). Wow! Then I got used to it, and now, anytime I'm back in France, I just find it unacceptable and wonder…
Earlier quoted context omitted.
Hey, for once my former life designing HA infrastructure is actually useful! The reality is that hardware HA was almost always terrible. Of the platforms you described: - Sun had no hardware HA ever, down to the unfathomable design that all of their E-class machines had only one power cord. They had SOME hotswap hardware, but the rules were byzantine - it couldn't be the first processor board, and it couldn't be the…
What about true HA hardware, for example like Tandem, or the systems on the Space Shuttle, etc.? A discussion of those systems from the 70s, 80s and 90s seems appropriate (but I am not knowledgeable enough to start it).
Both Tandem and Space Shuttle Flight Computer relied highly on software support for HA.
And it is my experience that most approaches to bolting on redundacy on system that is originally designed to run on single node are actually detrimental to the resulting availability. Either because it introduces additional failure modes that have to do with the fail-over mechanism itself (too eager failover, various byzantine generals failures of the control layer...) or because it depends on global consistency assumptions that are not in fact met (case in point: one high-reliability soft-realtime system with redundant application CPUs I was involved with is built with the assumption that you can reboot the application CPU at any time and it will catch up with system state as long as at least one other CPU is still running correctly. This ends up to not be a case as there is initialization phase that has to happen for the CPU to know the overall system state which interferes with the overall system operation as it involves injecting test patterns into global system inputs.)
Earlier quoted context omitted.
Your comment leaves the impression that IBM mainframes, not to mention high-end Unix systems from Sun, IBM, and HP, commonly had problems that would crash the machines and had no better availability than my desktop or perhaps a $3K server. Perhaps it was meant as hyperbole, but it would be interesting and valuable to hear some of the nuance from someone with your experience. IME talking to people, many, especially ma…
> it's hard to remember a box going down unless I commanded it to This could be a form of the law-of-large-numbers at work. If each customer only has a single box, and the failure rate is .1%, then 999 customers will never have a failure to remember, compared to the 1 who will. What does it look like if each customer has 1000 boxes?
Earlier quoted context omitted.
> it's hard to remember a box going down unless I commanded it to This could be a form of the law-of-large-numbers at work. If each customer only has a single box, and the failure rate is .1%, then 999 customers will never have a failure to remember, compared to the 1 who will. What does it look like if each customer has 1000 boxes?
I've managed far more than one box, and far more desktops/laptops than servers. The servers have been far more reliable.
I'm talking about the mainframe (or other high-end server) situation.
Granted, you were specifically referring to "mid-range Compaq/HP servers", so perhaps I didn't understand what mid-range meant.
If they're just a branded/enterprise version of commodity servers, I'm missing how those are representative of hardware HA of the kind that the OC finds almost always terrible.
If they're non-commodity, then I'm curious what their price tags are and if "far more than one box" equates to at least several hundred (else there's still a decent enough chance of not getting bitten by even a 1% failure rate).
The opening about availability and our expectations hits right at home. I'm originally from France, where stores closing during lunch time, sundays and mondays all day is normal. I remember when I moved to Canada, it seemed so amazing that stores were opened 7/7, and sometimes until 9pm (on Thursdays and Fridays). Wow! Then I got used to it, and now, anytime I'm back in France, I just find it unacceptable and wonder…
Because they are all closed at the same time?
Hmm, I think potted history is more correct... There is no mention hardware here. HP, Compaq, IBM and Sun all produced hardware with HA ability. Meaning that normal software could be run on two or more nodes, and should one break, it'd fall over with no loss of data, or outages. Here is a (contrived) video where they literally blow up a server stack: https://www.youtube.com/watch?v=qMCHpUtJnEI You can do this with VM…
The part I don't understand is how can the primary wait for the secondary to execute the same cpu instruction before producing output? this would be extremely slow even with very good network latency.
Also it seem in vsphere6 VMware have rewritten how FT is handled, dropping the old “lock-step” approach in favour of a new “fast check-pointing” type method.
If the memory content on the secondary is not exactly the same as what was on the primary before the crash. Then there is some kind of small dataloss.
This is the kind of thing I assumed could only be solved by having the primary run a consensus protocol like Raft or Paxos for each network request it receive before producing response to the request.
Hmm, I think potted history is more correct... There is no mention hardware here. HP, Compaq, IBM and Sun all produced hardware with HA ability. Meaning that normal software could be run on two or more nodes, and should one break, it'd fall over with no loss of data, or outages. Here is a (contrived) video where they literally blow up a server stack: https://www.youtube.com/watch?v=qMCHpUtJnEI You can do this with VM…
Hey, for once my former life designing HA infrastructure is actually useful! The reality is that hardware HA was almost always terrible. Of the platforms you described: - Sun had no hardware HA ever, down to the unfathomable design that all of their E-class machines had only one power cord. They had SOME hotswap hardware, but the rules were byzantine - it couldn't be the first processor board, and it couldn't be the…
Earlier quoted context omitted.
What about true HA hardware, for example like Tandem, or the systems on the Space Shuttle, etc.? A discussion of those systems from the 70s, 80s and 90s seems appropriate (but I am not knowledgeable enough to start it).
The hardware failures that hardware-based HA systems solve without software assistance are extremely unlikely on any modern platform (eg. CPU failure). In this regard it is somewhat remarkable, that original Pentium supports glueless(!) high-reliability redundant CPU mode (2 cores execute same instruction stream and you get NMI when the results diverge). It is somewhat telling that probably world's first 2 votes out…
Earlier quoted context omitted.
Your comment leaves the impression that IBM mainframes, not to mention high-end Unix systems from Sun, IBM, and HP, commonly had problems that would crash the machines and had no better availability than my desktop or perhaps a $3K server. Perhaps it was meant as hyperbole, but it would be interesting and valuable to hear some of the nuance from someone with your experience. IME talking to people, many, especially ma…
> it's hard to remember a box going down unless I commanded it to This could be a form of the law-of-large-numbers at work. If each customer only has a single box, and the failure rate is .1%, then 999 customers will never have a failure to remember, compared to the 1 who will. What does it look like if each customer has 1000 boxes?
Your mid sized mainframe will be at least a full rack, plus ancillaries. This can be analogous to a rack full of blades.
Earlier quoted context omitted.
> it's hard to remember a box going down unless I commanded it to This could be a form of the law-of-large-numbers at work. If each customer only has a single box, and the failure rate is .1%, then 999 customers will never have a failure to remember, compared to the 1 who will. What does it look like if each customer has 1000 boxes?
Remember that a "box" is a unit for commodity. Your mid sized mainframe will be at least a full rack, plus ancillaries. This can be analogous to a rack full of blades.
I'd argue that blades are already bordering on being outside of "commodity", at least in this context reliability, since a single chassis replaces some number of actually-commodity standalone boxes with both shared PoFs and highly-customized designs (especially high density that results in thermal unreliability if not outright failure).
[1] Although some blade systems did offer things like integrated shared switches, they failed to make them cheap enough, so, in practice, it didn't happen. Personally, I suspect there was never enough advantage to putting ethernet or even InfiniBand on a backplane instead of cables, not at blade chassis scale of a dozen nodes.