Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Client asks for 100% uptime

serverfault.com

Post reply on HN
Print view View original

41–50 of 100 posts

Re: Client asks for 100% uptime

#41

Post by runako »

All the posters are stuck on the fact that 100% availability is impossible. But why not instead try to learn from others who offer 100% availability, like Rackspace and SoftLayer? These (legitimate) providers know 100% availability is not possible, but they guarantee it anyway. How can they get away with this? Easy, they have a contractual SLA that indicates what their clients are entitled to when their network fails for any period of time. Further, neither is a low-cost provider, which allows them to engineer their systems to reduce incidences when clients will invoke the SLA.

Note that this doesn't mean that Rackspace is shady because they promise 100% knowing they can't deliver it. After all, they put their money where their mouth is! They have an incentive to actually achieve 100% uptime. I'm sure there are other applications where the target is 100% (not 5 nines) availability, especially in finance, medicine, and militaries.

My recommendation would be to take your engineering hat off, replace it with a business hat, and provide them with a series of price quotes for various uptime SLAs. And then make sure you're pricing high enough that when something goes down for any period of time that you can make good on your obligations under the SLA without losing too much sleep. Then let the client choose the SLA that matches their business needs and budget.

Re: Client asks for 100% uptime

#42

Post by snorkel »

oconnore wrote: post #26

I don't understand what the issue is. The client wants you to plan for disaster, and they aren't math oriented, so asking for 100% probability sounds reasonable. The engineer, as engineers are prone to do, remembered his first day of prob&stat 101, without considering that the client might not. When they say this, they aren't thinking about nuclear winter, they are thinking about Fred dumping his coffee on the office…

Most clients don't understand the R2D2 talk. They understand money, features, bugs, and downtime. So I've always explained it like so:

Uptime beyond 95% costs lots and lots of money. Magnitudes of order of more money. It requires redundant equipment, engineering all of the automatic failovers at every layer, lots of monitoring, and 24/7 technical staff to watch everything like a hawk. Not ... in ... your ... budget.

... or you could rest in the comfort of knowing that services like Twitter have achieved mammoth success despite long and embarrassing outages. I thought you might you see it that way. Good choice.

Re: Client asks for 100% uptime

#43

Post by shabble »

gaius wrote: post #15
joelhaasnoot wrote: post #5

Earlier quoted context omitted.

Public transport does this too: in order to provide a robust, perfect implementation of a schedule, you need extra busses/trains/street cars (expensive capital goods) and extra manhours (generally most expensive part of the operation). Rather than invest too much money in making sure the schedule can be met, it's cheaper to pay fines when there are delays.

British Rail once declared they operated on a 66-minute hour for this reason.

A (garishly coloured) reference: http://www.lococarriage.org.uk/66minute.htm

I've actually had quite good experiences claiming ticket credits or total refunds for some long-distance UK train journeys. I can't find the exact terms, and they potentially vary per-operator, but it's around 50% refund for up to 30 minutes delay, and a full ticket cost refund if it's >60 minutes. A couple of times, I've had a trip delayed by 50-59 minutes. I suspect this isn't a coincidence.

Re: Client asks for 100% uptime

#44

Post by hmottestad »

So 100% uptime is really difficult to achieve, hardware wise. Software wise you'll have to prove that there are no bugs in the system that might bring it down. That is much, much, harder.

You can have 100 servers in 100 different countries and have the client automatically change to another server if the one they are connected to goes down. But if there is a software bug that crashes all your clients on start-up, or worse, crashes all your servers (think what happened to Skype not long ago).

Also, never underestimate bugs in hardware (pentium 1). You'll need multiple locations, multiple hardware, multiple operating systems, multiple compilers, multiple versions of the software.... Standardizing on one of these components may bring down your entire system!

Re: Client asks for 100% uptime

#46

Post by kahawe »

I see several possible approaches, if you really want to have that client.

This easiest would be to just talk to them, try to find out what that "100%" is actually REALLY all about and make them understand that from a technical point of view, 100% will add a lot of things to the project budget. A "100%" demand in a smaller project for a typical small-to-medium business will likely mean something different than "100%" in a project for the NYSE. So, talk to the customer and find out what it is actually all about and then plan and quote according to their actual needs. So, this makes it more a requirements-engineering type of problem, not necessarily a hacker problem.

Or you just say "yes, of course" and tell them how super reliable the system is and then let the guys in legal work it out in the fine print and cover your ass.. but don't expect much happiness and continued business from that client then once they find out what's going on.

But, in a more honest approach, maybe this is actually all they really want and need? Maybe it actually is enough for them to have someone to blame and pay some penalties for violating SLAs. Again, you need to find that out.

Not a typical hacker-hacker-problem but surely an issue a hacker would typically encounter, even on a daily basis, and should learn to deal with.

Re: Client asks for 100% uptime

#47

Post by oconnore »

SoftwareMaven wrote: post #40
oconnore wrote: post #26

I don't understand what the issue is. The client wants you to plan for disaster, and they aren't math oriented, so asking for 100% probability sounds reasonable. The engineer, as engineers are prone to do, remembered his first day of prob&stat 101, without considering that the client might not. When they say this, they aren't thinking about nuclear winter, they are thinking about Fred dumping his coffee on the office…

This is the answer missing from SF. This is the true "what would it take" that could be presented back to the client. Would the client go for the cost? Who knows,but that's the client's decision. Add a little more data on the DNS failover and it would be a good community wiki entry there.

I'm flattered. I don't have a SF account but feel free to copy/paste if you think they would enjoy reading it.

Re: Client asks for 100% uptime

#48

Post by Retric »

oconnore wrote: post #31
brown9-2 wrote: post #28

Earlier quoted context omitted.

But "three 9 reliability" is still not the same thing as 100%. The contractor has a right to be concerned about the 100% figure making it's way into a contract.

No, three 9 reliability for a single server. 1-.001^3 = .999999999, which is under a second expected per year, which the client will never notice even with good monitoring tools, and therefore will never invoke the contract.

Your assuming independence to a level that does not exist. Consider a Y2K style bug in the OS would could take down all severs for an extended period of time. Or someone could write a virus that uses a zero day exploit etc.

Re: Client asks for 100% uptime

#49

Post by DanBC »

Helping your customers understand what they actually want to buy is part of selling, surely? Things are made trickier by PHBs in the client company claiming that everything is mission critical and that they can never ever have any downtime ever for any reason. Educating these people about, for example, just how flaky email and dns can be is important for your sanity.

See, for example, these couple of posts from a Microsoft public newsgroup ten years ago: (https://groups.google.com/group/microsoft.public.backoffice....) (https://groups.google.com/group/microsoft.public.backoffice....)

Some customers are clueless, but at least they care about the data.

Re: Client asks for 100% uptime

#50

Post by twstdroot »

marquis wrote: post #23
illumin8 wrote: post #21

Look at F5 networks Global Traffic Manager. It's really just a fancy DNS server. You set your TTL (time to live) down to just a few seconds and it monitors your main and standby sites. If one of the sites goes down it changes your A records to point to the new site. It can even do load balancing across sites based on response time or number of connections. They are expensive, but this is how large companies like Yaho…

This is the kind of service I'd love to attack as a side project, it's fascinating. Though I'm sure someone out there reading this, has something like this service but affordable for startups?

Several DNS hosting services do this at varying costs. DurableDNS (which I founded but have sold) does it at a low cost. It's fairly trivial softwarewise as long as you have the redundant hardware, DCs, etc.
Post reply on HN