Client asks for 100% uptime • Hacker News Board

Re: Client asks for 100% uptime

#61

Post by rickmb » Fri, Sep 30, 2011, 4:17 PM UTC

duck wrote: post #51

msy wrote: post #2

It's always helpful when clients give unambiguous signs of unreasonable insanity upfront instead of hiding it until you're halfway through the project. It makes running away as far and fast as humanly possible so much easier.

There also was this comment on the SO thread: I would personally RUN from this client as fast as possible. I suspect this won't be the last crazy idea they may have (from a technology standpoint). Why run though? They probably just don't understand what 100% means and it just takes you explaining it to them. Or simply state that you cannot meet that requirement and see if they still want you to bid on the project.

You've just quoted the reason why: it won't be the last crazy idea they may have.

That's pretty much an absolute certainty. Even if you can convince them with reasonable arguments to accept a few point less uptime, you're going to be having the same kind of discussion many times after on different subjects.

You have to be really, really sure you want and need this kind of client. Most of the time (around 100%...) they are more trouble than they are worth.

Re: Client asks for 100% uptime

#62

Post by mkup » Fri, Sep 30, 2011, 4:33 PM UTC

rwmj wrote: post #11

Come on, this is possible. First we're going to have to get the governments of the world together to agree to remove all nuclear weapons. Second would be getting that asteroid tracking and deflection system working. Quantum physics does unfortunately predict that the earth might flick out of existence with some small probability, but by distributing the website across the universe we can reduce this probability arbit…

And what about gamma ray bursts from nearby supernova directed to the Earth?

Re: Client asks for 100% uptime

#63

Post by snorkel » Fri, Sep 30, 2011, 4:52 PM UTC

swombat wrote: post #56

snorkel wrote: post #42

Earlier quoted context omitted.

Most clients don't understand the R2D2 talk. They understand money, features, bugs, and downtime. So I've always explained it like so: Uptime beyond 95% costs lots and lots of money. Magnitudes of order of more money. It requires redundant equipment, engineering all of the automatic failovers at every layer, lots of monitoring, and 24/7 technical staff to watch everything like a hawk. Not ... in ... your ... budget.…

First of all, uptime of 95% means 18 and a quarter entire days of downtime per year. That's horrendous. I wouldn't host my dog's website on a server with that kind of SLA - and I don't even have a dog. Secondly, although Twitter got away with large helpings of downtime, that doesn't mean that every business type can. Twitter is not (or at least was not, for most of its existence) business-critical to anyone. If Twitt…

Two problems:

The definition of "uptime", the host SLAs only covers network uptime and environment uptime, but clients consider "uptime" to mean application level uptime, which includes downtimes for server maintenance, steady-state backups, backup restores, deploying new releases, etc ... anything other than service is not 100% fully functional = downtime in their minds.

Also for costs, "reasonably fast failover to another similar host" implies live redundant equipment at another host which is double the hosting costs, that's a big pill to swallow, so big that most orgs would rather suffer the downtime when they see the real cost of full redundancy.

Re: Client asks for 100% uptime

#64

Post by Symmetry » Fri, Sep 30, 2011, 4:53 PM UTC

oconnore wrote: post #26

I don't understand what the issue is. The client wants you to plan for disaster, and they aren't math oriented, so asking for 100% probability sounds reasonable. The engineer, as engineers are prone to do, remembered his first day of prob&stat 101, without considering that the client might not. When they say this, they aren't thinking about nuclear winter, they are thinking about Fred dumping his coffee on the office…

Maybe it would be good to write the custom an email explaining stuff like

  99%=Well run server
  99.9%=Multiple backups, will cover most hardware failures
  99.99%=Top grade commercial 
  99.999%=What companies like Google or Yahoo can achieve
  99.9999%=Hopefully the US strategic defense systems are this reliable

Also, you're assumption about the servers are entirely independant. That's a reasonable assumption in terms of fires and blackouts and floods, but not for software problems. You really can't assume that unless the servers are all running entirely different software stacks on different operating systems.

Re: Client asks for 100% uptime

#65

Post by run4yourlives » Fri, Sep 30, 2011, 5:03 PM UTC

snorkel wrote: post #42

oconnore wrote: post #26

I don't understand what the issue is. The client wants you to plan for disaster, and they aren't math oriented, so asking for 100% probability sounds reasonable. The engineer, as engineers are prone to do, remembered his first day of prob&stat 101, without considering that the client might not. When they say this, they aren't thinking about nuclear winter, they are thinking about Fred dumping his coffee on the office…

Most clients don't understand the R2D2 talk. They understand money, features, bugs, and downtime. So I've always explained it like so: Uptime beyond 95% costs lots and lots of money. Magnitudes of order of more money. It requires redundant equipment, engineering all of the automatic failovers at every layer, lots of monitoring, and 24/7 technical staff to watch everything like a hawk. Not ... in ... your ... budget.…

I'm getting better than 95% up time on my home network. If you told me that and I was your client, I'd be going elsewhere.

Over 99% costs lots of money, yes. How much is dependent on how close to 100 you are looking to get, but that's the client's decision. 99% though is a perfectly acceptable standard.

Re: Client asks for 100% uptime

#66

Post by droithomme » Fri, Sep 30, 2011, 5:13 PM UTC

You can do this, but I suspect they might not like the cost quote of $10 trillion per year, and they 10 years of lead time it will cost you to build a worldwide network of secure underground bunkers with their own air, water, food and energy supplies, and a hardened shadow internet that duplicates the function of the existing internet.

Re: Client asks for 100% uptime

#67

Post by samuel » Fri, Sep 30, 2011, 5:16 PM UTC

brador wrote: post #10

samuel wrote: post #8

I would answer that Price is a function of uptime given by the following formula: Price(uptime) = K*(1/1 - uptime) so I would ask for infinite dollars...

I think you mean Price = K/(1-uptime)

(Embarrased) Yes of course it is. And that's coming from someone who presumes to know operator precedence and despises programmers who use extra parens for no good reason.

Re: Client asks for 100% uptime

#68

Post by oconnore » Fri, Sep 30, 2011, 5:32 PM UTC

shabda wrote: post #60

oconnore wrote: post #58

Earlier quoted context omitted.

I am making no such assumption. See [1] in my original post. I already talked about intersection. Feel free to add Y3K to the list of nuclear war, chinese hackers, etc. The intersection is incredibly small, and not something that I am going to include in my back-of-the-napkin calculation.

Your failover code never has bugs? See the linked discussion, only 3 out of 20 top sites had 5 nines.

For enough time and money dumped into code auditing and hiring smart people, no. Not that most companies should do that, but it is possible if you want to pay for it. Most companies (rightly) prioritize innovation, scalability and profit margins over absolute reliability.

How many of those top sites actually prioritized reliability? Is it even justifiable for their business models? I bet you can find a lot better reliability engineering in bank, credit, and stock systems. For example, when was the last time the Visa credit network crashed (as a whole, not localized outages)? Nasdaq?

Re: Client asks for 100% uptime

#69

Post by timwiseman » Fri, Sep 30, 2011, 5:48 PM UTC

run4yourlives wrote: post #65

snorkel wrote: post #42

Earlier quoted context omitted.

Most clients don't understand the R2D2 talk. They understand money, features, bugs, and downtime. So I've always explained it like so: Uptime beyond 95% costs lots and lots of money. Magnitudes of order of more money. It requires redundant equipment, engineering all of the automatic failovers at every layer, lots of monitoring, and 24/7 technical staff to watch everything like a hawk. Not ... in ... your ... budget.…

I'm getting better than 95% up time on my home network. If you told me that and I was your client, I'd be going elsewhere. Over 99% costs lots of money, yes. How much is dependent on how close to 100 you are looking to get, but that's the client's decision. 99% though is a perfectly acceptable standard.

There is a big difference between getting >95% up time and being able to actually back up an SLA for > 95% up time.

Re: Client asks for 100% uptime

#70

Post by rwmj » Fri, Sep 30, 2011, 5:48 PM UTC

mkup wrote: post #62

rwmj wrote: post #11

Come on, this is possible. First we're going to have to get the governments of the world together to agree to remove all nuclear weapons. Second would be getting that asteroid tracking and deflection system working. Quantum physics does unfortunately predict that the earth might flick out of existence with some small probability, but by distributing the website across the universe we can reduce this probability arbit…

And what about gamma ray bursts from nearby supernova directed to the Earth?

The Dyson sphere surrounding the solar system is obviously going to be expensive, but what's money when you want 100% uptime?