341–350 of 646 posts
Earlier quoted context omitted.
> What am I missing? My guess is that they cheaped out in having redundant PSUs to get you to use multiple availability zones. (More zones = more revenue) Even a single PSU shouldn’t be an issue if they plugged in an ATS switch though.
Unless the ATS breaks, which happens.
Earlier quoted context omitted.
I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…
I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.
If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ?
Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available.
No point in investing in S3 and then doing it again yourself.
If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…
So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…
If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…
So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…
Earlier quoted context omitted.
another example of a single dc in a single AZ rendering an entire region almost unusable. This has shades of eu-central-1 all over again.
Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?
"almost unusable" is maybe exaggerating, but there were definitely issues affecting more than just the single AZ.
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
Earlier quoted context omitted.
Human capital side would disagree with that I think. You're assuming the organization which owns this small/medium web app has the personnel already on staff to handle such a thing. If you're outsourcing that, you'd likely have to pay a boatload just for someone to be available for help, let alone the actual tasks themselves. Like you said, if you're on-prem and something goes down, you can do something. But you've g…
> Human capital side would disagree with that I think I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure.
Bare metal infrastructure requires a lot more management at any given scale. I mean, you can run stuff that lets you do part of the management the same as cloud resources, but you also have to then manage that software and manage the hardware.
AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…
100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down. edit: of course, AWS does have this: AWS Fault Injection Simulator
AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…
100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down. edit: of course, AWS does have this: AWS Fault Injection Simulator
Earlier quoted context omitted.
A lot of people will automatically fail over jobs to other AZ's. That often involves spinning up lots more EC2 instances and moving PB's of data. The end result is all capacity on other AZ's gets used up, and networks get full to capacity, and even if those other zones are technically working, practically they aren't really usable.
While there may be more machines provisioned, many orgs run active setups for failover so they aren’t as affected. In terms of data transfer, it should already be there. Where would it come from? Certainly not the dead AZ.