Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Interuptions

news.ycombinator.com

Post reply on HN
Print view View original

51–56 of 56 posts

Re: AWS Service Interuptions

#51

Post by jdc0589 »

sidcool wrote: post #41
karmacondon wrote: post #23

I recently switched to Google Compute Engine. It's cheaper and so far more reliable than AWS. Might be another option for some people here.

I am trying to convince people at my work to move to GCP from AWS, but AWS truly has become the Microsoft of Cloud computing. Many people have no idea there are other providers like GCP, Azure, DigitalOcean etc.

Azure might be great in a year or so, but it makes me uneasy as is. Some of the services are great, but a lot of them are pretty fragmented. I've had so many instances where our billing/usage data has just "disappeared" for a few days, undocumented changes have been made to the formats of reports/exports/APIs, and official documentation is plain wrong that I just can't recommend Azure to anyone. Not to mention they have the most expensive infrastructure costs out of the major players (even with an EA and decent monetary commitment); their premium for Windows licensing is the lowest by far though (not surprising), so it does end up being a cheaper option for super windows heavy shops though.

Re: AWS Service Interuptions

#52

Post by mrmondo »

mbesto wrote: post #42
mrmondo wrote: post #28

Earlier quoted context omitted.

Yeah we monitor lots of Amazon & Microsoft 'cloud' services, we observe much, much higher downtime / number of outages than they ever report in a order of 50 to 1 or more. What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime and that they're not only a good option but t…

> What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime Their uptime is much higher on average than any IT team I've ever been involved in.

Oh wow really? That's really bad - you must have worked with some really poor ops teams in the past. Last year we measured less than 97% uptime on AWS Sydney, and a shocking 96% uptime for Office 365 exchange online. Most of the problems when we investigated them out of interest were due to either internet network routing issues within their networks (or first ISP hop), or they just had hosts outright fail. The 'cloud' is just outsourced hardware with a provided toolset (APIs etc...), Amazon itself claims that you must have your hosts across various zones to get decent uptime - that's like saying "oh yes - the Toyota Carolla is really reliable, it works 99.99% of the time... As long as you buy a second one for when it's not available".

Our internal uptime is 99.985 in production, we are fast moving and roll out changes every day, we run mainline kernels and all of our 350 odd servers and 800~ containers are running on completely vendor independent, open source software.

I'm not saying it's easy, but the middle man is there to help you if you can't find or afford up front good operational engineers, or to take your money because their advertising has made you believe that they are always the best decision.

We perform an in-detail yearly cross-cost comparison between AWS and our operated datacentre, the cost to run and maintain the same uptime, processing power (and yes we take into account spinning down instances at night etc...), bandwidth between zones, backups and customers and it really hasn't improve at all over the past 3 years. This year the review came back that our yearly expenditure on operational expenses would increase from approximately $500,000 (including human resources) to well over $3,000,000 a year. (Not kidding), the margin of error was approximated at between 10-20%.

Re: AWS Service Interuptions

#53

Post by mbesto »

mrmondo wrote: post #52
mbesto wrote: post #42

Earlier quoted context omitted.

> What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime Their uptime is much higher on average than any IT team I've ever been involved in.

Oh wow really? That's really bad - you must have worked with some really poor ops teams in the past. Last year we measured less than 97% uptime on AWS Sydney, and a shocking 96% uptime for Office 365 exchange online. Most of the problems when we investigated them out of interest were due to either internet network routing issues within their networks (or first ISP hop), or they just had hosts outright fail. The 'clou…

> Oh wow really? That's really bad - you must have worked with some really poor ops teams in the past.

You sound genuinely very smart and knowledgeable in this area. But the other 90% of the workers in this sector are not.

> Amazon itself claims that you must have your hosts across various zones to get decent uptime - that's like saying "oh yes - the Toyota Carolla is really reliable, it works 99.99% of the time... As long as you buy a second one for when it's not available".

Wait, you don't have a second data center for your mission critical systems in case your primary fails?

> We perform an in-detail yearly cross-cost comparison between AWS and our operated datacentre...bandwidth between zones, backups and customers and it really hasn't improve at all over the past 3 years

I totally agree. If you have the right resources, a good data center partner and well defined process, then "the cloud" isn't for you. For the other 90% of the people out there that simply don't have the know-how, knowledge, or resources to find talented IT operational excellence, then AWS totally makes sense.

Re: AWS Service Interuptions

#54

Post by mrmondo »

mbesto wrote: post #53
mrmondo wrote: post #52

Earlier quoted context omitted.

Oh wow really? That's really bad - you must have worked with some really poor ops teams in the past. Last year we measured less than 97% uptime on AWS Sydney, and a shocking 96% uptime for Office 365 exchange online. Most of the problems when we investigated them out of interest were due to either internet network routing issues within their networks (or first ISP hop), or they just had hosts outright fail. The 'clou…

> Oh wow really? That's really bad - you must have worked with some really poor ops teams in the past. You sound genuinely very smart and knowledgeable in this area. But the other 90% of the workers in this sector are not. > Amazon itself claims that you must have your hosts across various zones to get decent uptime - that's like saying "oh yes - the Toyota Carolla is really reliable, it works 99.99% of the time... A…

Yes we have two datacentres and we do have a few VPS mostly for triangulation of monitoring, but honestly, in four years - we haven't had to failover once, although we practise it with our applications almost every single day.

Thank you for the kind words there, I think one major thing for us is that we've hired a small number of just the right people, each with quite different backgrounds and we work VERY closely with our developers. Every bit of configuration is kept in GIT and we CI / CD whatever we can.

Re: AWS Service Interuptions

#55

Post by mbesto »

mrmondo wrote: post #54
mbesto wrote: post #53

Earlier quoted context omitted.

> Oh wow really? That's really bad - you must have worked with some really poor ops teams in the past. You sound genuinely very smart and knowledgeable in this area. But the other 90% of the workers in this sector are not. > Amazon itself claims that you must have your hosts across various zones to get decent uptime - that's like saying "oh yes - the Toyota Carolla is really reliable, it works 99.99% of the time... A…

Yes we have two datacentres and we do have a few VPS mostly for triangulation of monitoring, but honestly, in four years - we haven't had to failover once, although we practise it with our applications almost every single day. Thank you for the kind words there, I think one major thing for us is that we've hired a small number of just the right people, each with quite different backgrounds and we work VERY closely wi…

> Yes we have two datacentres

That's all that Multi-AZ is mate ;)

Re: AWS Service Interuptions

#56

Post by sidcool »

jdc0589 wrote: post #51
sidcool wrote: post #41

Earlier quoted context omitted.

I am trying to convince people at my work to move to GCP from AWS, but AWS truly has become the Microsoft of Cloud computing. Many people have no idea there are other providers like GCP, Azure, DigitalOcean etc.

Azure might be great in a year or so, but it makes me uneasy as is. Some of the services are great, but a lot of them are pretty fragmented. I've had so many instances where our billing/usage data has just "disappeared" for a few days, undocumented changes have been made to the formats of reports/exports/APIs, and official documentation is plain wrong that I just can't recommend Azure to anyone. Not to mention they h…

Although I am quite optimistic about Azure, GCP seems like the best bet at the moment. I think of factors like Reliability, performance, availability, cost and longevity.
Post reply on HN