Live data from Hacker News

Client asks for 100% uptime

serverfault.com

21–30 of 100 posts

Re: Client asks for 100% uptime

#21
Look at F5 networks Global Traffic Manager. It's really just a fancy DNS server. You set your TTL (time to live) down to just a few seconds and it monitors your main and standby sites. If one of the sites goes down it changes your A records to point to the new site. It can even do load balancing across sites based on response time or number of connections.

They are expensive, but this is how large companies like Yahoo keep close to 100% uptime.

Explain to the customer that even with a hot site, the failover can take a few minutes. Also, some ISPs don't honor TTL and cache DNS queries for longer than they should. The Internet isn't perfect, and usually each extra 9 you add is around 4x more cost.

Re: Client asks for 100% uptime

#22
post #18

Their craziness doesn't matter. Usually crazy customers aren't rich. So if you build to their craziness, you'll lose the customer. You need to build an appropriate infrastructure that will win the bid, figure out what you can achieve (99.9%/99.99% uptime) and build in enough overhead to cover your SLA penalties. Or negotiate a monitoring methodology that is in your favor. (ie. exclude planned maintenance windows, use…

> Usually crazy customers aren't rich.

On a personal level, I think individual people who've become wealthy have a more reasonable outlook when it comes to stuff like this, but this doesn't apply to rich _companies_.

The least reasonable clients I've ever had were employees of large companies, and more specifically those who'd been recently empowered with the responsibility of the project I was working on and lacked the perspective required to realize that crazy doesn't help anyone.

It's not their money, but it's their decision, and that's a petri dish for crazy.

Re: Client asks for 100% uptime

#23

Look at F5 networks Global Traffic Manager. It's really just a fancy DNS server. You set your TTL (time to live) down to just a few seconds and it monitors your main and standby sites. If one of the sites goes down it changes your A records to point to the new site. It can even do load balancing across sites based on response time or number of connections. They are expensive, but this is how large companies like Yaho…

This is the kind of service I'd love to attack as a side project, it's fascinating. Though I'm sure someone out there reading this, has something like this service but affordable for startups?

Re: Client asks for 100% uptime

#25

Look at F5 networks Global Traffic Manager. It's really just a fancy DNS server. You set your TTL (time to live) down to just a few seconds and it monitors your main and standby sites. If one of the sites goes down it changes your A records to point to the new site. It can even do load balancing across sites based on response time or number of connections. They are expensive, but this is how large companies like Yaho…

Alternatively look at Zeus's Traffic Manager and Global Load Balancer.

http://www.zeus.com/products/traffic-manager

http://www.zeus.com/products/global-load-balancer

These are smart, easy to use and well supported. They're also to be found inside big cloud offerings like AWS and Joyent.

Re: Client asks for 100% uptime

#26
I don't understand what the issue is. The client wants you to plan for disaster, and they aren't math oriented, so asking for 100% probability sounds reasonable. The engineer, as engineers are prone to do, remembered his first day of prob&stat 101, without considering that the client might not.

When they say this, they aren't thinking about nuclear winter, they are thinking about Fred dumping his coffee on the office server, a disk crashing, or an ISP going down.

Furthermore, you can accomplish this. With geographically distinct, independent, self monitoring servers, you will basically have no downtime. With 3 servers operating at an independent[1] three 9 reliability, with good failover modes, your expected downtime is under a second per year [2]. Even if this happens all at once, you are still within a reasonable SLA for web connections, and therefore the downtime practically does not exist.

The client still has to deal with doomsday scenarios, but Godzilla excluded, he will have a service that is "always" up.

[1] A server in LA is reasonably independent from the server in Boston, but yes, I understand that there is some intersection involving nuclear war, Chinese hackers crashing the power grid, etc. I don't think your client will be upset by this.

[2] DNS failover may add a few seconds. You are still in a scenario where the client has to retry a request once a year, which is, again, within a reasonable SLA, and not typically considered in the same vein as "downtime". With an application that automatically reroutes to an available node on failure, this can be unnoticeable.

Re: Client asks for 100% uptime

#27
100% uptime is unrealistic for big companies because at scale, it costs a lot. For example, replicating is expensive with transmission, storage and maintenance costs.

When the Amazon incident happened, I did an analysis and found that the cost about triples if stored in an external data centre. Almost 6x cost if hosting overseas despite same company.

I then understood why companies like Reddit do not aim for 100% uptime possible beyond the data centre. It depends on how much the customer (or client) is willing to pay that determines the uptime aim (I think Reddit's aim for example is 90% at least).

Re: Client asks for 100% uptime

#28

I don't understand what the issue is. The client wants you to plan for disaster, and they aren't math oriented, so asking for 100% probability sounds reasonable. The engineer, as engineers are prone to do, remembered his first day of prob&stat 101, without considering that the client might not. When they say this, they aren't thinking about nuclear winter, they are thinking about Fred dumping his coffee on the office…

But "three 9 reliability" is still not the same thing as 100%. The contractor has a right to be concerned about the 100% figure making it's way into a contract.
Post reply on HN