Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

431–440 of 646 posts

Re: Tell HN: AWS appears to be down again

#431

Post by hayd »

aeonflux wrote: post #311
kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Having an additional AWS account which some S3 backs up to, with write only permissions (no delete) and in an account that is not used by anyone, seems like a good idea for this type of situation/concern.

Re: Tell HN: AWS appears to be down again

#432

Post by thedougd »

jmartrican wrote: post #310
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Seems like multi-cloud solution might be the way to go.

I doubt it. The complexity of multi-cloud will also give you downtime.

Most of the folks impacted by cloud outages do not have highly available systems in place. Perhaps, for their business, the cost doesn't justify the outcome.

If you need high uptime for instances, build your system to be highly available and leverage the fault domain constructs your provider offers (placement groups, availability zones, regions, load balancing, DNS routing, autoscaling groups, service discovery, etc). For instances, double down and use spot instance and maximum lifetimes in your groups so that you're continuously validating your application can recovery from instance interruptions.

If you're heavy on applications that leverage cloud APIs, such as is often the case with labmdas, then strongly consider multi-region active/active as API outages tend to cross AZ's and impact the entire region.

Re: Tell HN: AWS appears to be down again

#433

Post by dylan604 »

Tempest1981 wrote: post #416
dylan604 wrote: post #384

Earlier quoted context omitted.

>And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All driv…

Would like to know the manufacturer and model.

Sorry, this was back in 2006-2007 time frame. I have no idea on model numbers as that's just not information I ever cared to commit to memory.

Re: Tell HN: AWS appears to be down again

#434

Post by spmurrayzzz »

electroly wrote: post #381
pm90 wrote: post #294

Earlier quoted context omitted.

> The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services. I think cloud providers…

AWS IAM is designed with a control plane / data plane dichotomy. Even if the control plane is completely dead and all API requests are failing, services in a steady-state (i.e. not responding to changes via API calls) can still rely on IAM roles using their cached information. For example, in the recent us-east-1 outage when you couldn't start new services because IAM checks would fail, existing EC2 instances that re…

I was gonna respond with the same commentary here. That has been my experience beyond just IAM controls and why I advocate for passive systems for critical workloads.

Sometimes this _can_ be costly. For example with something like autoscaling, thats an active system I've seen fail when seemingly unrelated systems are failing. The result is scaling out systems intentionally ahead of time to deal with oversubscription or burst traffic which can leave you with (costly) idle compute.

I don't mind this tradeoff personally, but can understand that budget constraints are going to be different org to org.

Re: Tell HN: AWS appears to be down again

#435

Post by tluyben2 »

tluyben2 wrote: post #420
AtNightWeCode wrote: post #414

Earlier quoted context omitted.

"Won't happen". The 40,000 hours of runtime bug did happen. I would recommend people to take backups and store them offline or at least isolated from the main storage.

Sure, I plan for it but I do not expect it. And it never happened for me over decades. But I did plan for it, just not in the way the parent said and it did cost me far less.

Good to know ;) still think I am lucky; have had very little harddisks fail across 1000s. And only very few unrecoverable (had to restore from backup) failures; those weren't HD failures but software failures; hds were fine.

Re: Tell HN: AWS appears to be down again

#436

Post by bcrosby95 »

dragonwriter wrote: post #347
Nextgrid wrote: post #322

Earlier quoted context omitted.

> Human capital side would disagree with that I think I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure.

> I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure. Bare metal infrastructure requires a lot more management at any given scale. I mean, you can run stuff that lets you do part of the management the same as cloud resources, but you also have to then manage…

Define "a lot".

We colocate about 20 servers and on the average month, no one spends any time managing them. At all.

Re: Tell HN: AWS appears to be down again

#437

Post by nfkrk8j »

aeonflux wrote: post #311
kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

No post body was provided.

Re: Tell HN: AWS appears to be down again

#438

Post by kburman »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

AWS or Google or any other reputable cloud provider are still far more better options then your local backup. Only way I see you losing your data is account getting locked.

Re: Tell HN: AWS appears to be down again

#439

Post by dkryptr »

biohax2015 wrote: post #349
dkryptr wrote: post #332

Earlier quoted context omitted.

100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down. edit: of course, AWS does have this: AWS Fault Injection Simulator

AWS Fault Injection Simulator does this.

TIL. Thank you!

Re: Tell HN: AWS appears to be down again

#440

Post by bob1029 »

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> I've never worked anywhere with Amazon's budget so why are they not handling this?

Perhaps we are going to discover how AWS produces such lofty margins by way of their next RCA publication.

Post reply on HN