41–50 of 56 posts
I recently switched to Google Compute Engine. It's cheaper and so far more reliable than AWS. Might be another option for some people here.
It will take a direct hit with nuclear weapon on the datacenter for Amazon to change icon to red on service status page.
Yeah we monitor lots of Amazon & Microsoft 'cloud' services, we observe much, much higher downtime / number of outages than they ever report in a order of 50 to 1 or more. What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime and that they're not only a good option but t…
Their uptime is much higher on average than any IT team I've ever been involved in.
It will take a direct hit with nuclear weapon on the datacenter for Amazon to change icon to red on service status page.
It will take a direct hit with nuclear weapon on the datacenter for Amazon to change icon to red on service status page.
Yeah we monitor lots of Amazon & Microsoft 'cloud' services, we observe much, much higher downtime / number of outages than they ever report in a order of 50 to 1 or more. What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime and that they're not only a good option but t…
You might have an incident affecting just 2% of the API calls, and affecting less than 2% of the user base (even that would be unusually large and a source of big drama internally). The service could be super stable and extremely reliable, but that 98% could get completely the wrong idea if they saw a service status, (and of course from a PR perspective, the same goes for anyone looking to use the platform.)
A service dashboard is an extremely blunt tool with which to pass out a message about service status. It renders what is an extremely nuanced situation down to "All good, maybe, no, DEAD"
To give a rough example, one service I was familiar with had a "page everyone in the team" level of incident. API availability tanked, badly. It looked atrocious, and seemed like hardly any requests were getting through successfully. You'd have every expectation that they should at least post a yellow alert, if not approaching red. It turned out that it was one single customer who's requests were failing (I forget the reason why), but due to a bug in the customer's software consuming the API, every time it got a 500 response, it would immediately resend the request, every single time, with no timeout or limited retry number. It reached such a terrific pace it got to the point where they made up a huge majority of all the requests hitting the endpoint. Every other customer using the service was completely fine. If you'd looked at the API graphs you'd think "POST YELLOW, POST YELLOW, NOW NOW NOW!", but because they took time to figure out the actual impact, they found out that would have been totally the wrong thing to do.
Service health dashboards are a neat idea, but one that is in desperate need of a rethink and overhaul. It has some value when you're a smaller service, but it just doesn't accurately scale with the platform.
I'm not sure what the real solution is. They've somehow got to pull together TB of logs and/or metrics to make an accurate assessment of the scenario, and do it in a matter of minutes, so as to provide accurate updates, and not needlessly panic customers.
Earlier quoted context omitted.
Indeed it is, it was a massive struggle getting approval to move into a cloud service in the first place.
I almost hate to point this out, then, but you did consider that there's no guarantee that an AZ is a single DC, right?
I recently switched to Google Compute Engine. It's cheaper and so far more reliable than AWS. Might be another option for some people here.
Earlier quoted context omitted.
Indeed it is, it was a massive struggle getting approval to move into a cloud service in the first place.
Really ? Pointing the local gov department's officer to AWS's IRAP compliance cert was all that was needed to move quite a lot of their stuff unto AWS.
The classic irony for me was a service manager in just such an environment resisting a cloud move "because it's someone else's computer" - even though his (ancient) application was running on a rented partition of a remote, IBM owned & operated S/390...
No surprise therefore that the big clouds have country resources dedicated to moving the needle on cloud awareness in highly regulated environments.
(obdisclosure: I am former .au AWS manager)
I realize that some systems may need to have all of their servers located close together in a single AZ. But barring that, if this took you offline, you should really consider spreading your instances across AZs. It's so easy there's no excuse not to do it. Another thing to look into is EC2 Auto Recovery [1]. I don't know if this would've kicked in with today's event, but it's worth setting up as an extra safety net.…
The site I manage is load balanced across both AZs ap-southeast-2a and ap-southeast-2b which did not save it. At the moment ec2 statuses are not being updated which is preventing ELBs from registering instances as healthy. Both AZs are directly under the deluge and I don't believe only one AZ is affected for a second. The size of the storm can be seen here http://www.bom.gov.au/products/IDR713.loop.shtml#skip
Earlier quoted context omitted.
Really ? Pointing the local gov department's officer to AWS's IRAP compliance cert was all that was needed to move quite a lot of their stuff unto AWS.
Yeah, and I'm curious about which sector or agency is the culprit here. Even APRA (the financial regulator) are cloud-friendly now, if you engage them at the start of an adoption process. My wild guess is health insurance, being a sector where IT is notoriously hidebound, but it could just be a case of overzealous/interfering/uncomprehending lawyers. A security policy that precluded cross-site service or data replica…
lawyers
Noone ever seems to be able to refer to a specific law, but then, it's an IT person talking to lawyers, so there are some battles you just don't fight.
Earlier quoted context omitted.
Really ? Pointing the local gov department's officer to AWS's IRAP compliance cert was all that was needed to move quite a lot of their stuff unto AWS.
Yeah, and I'm curious about which sector or agency is the culprit here. Even APRA (the financial regulator) are cloud-friendly now, if you engage them at the start of an adoption process. My wild guess is health insurance, being a sector where IT is notoriously hidebound, but it could just be a case of overzealous/interfering/uncomprehending lawyers. A security policy that precluded cross-site service or data replica…