Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

611–620 of 646 posts

Re: Tell HN: AWS appears to be down again

#611

Post by pbecotte »

reaperducer wrote: post #545
BenjiWiebe wrote: post #540

Earlier quoted context omitted.

Are you sure about the few minutes part? The standby generators I've seen take seconds to go from off to full load. We have an 80kw model, but I've also seen videos of load tests of much larger generators and they also take only seconds to go to full load.

It might depend on when the backup system was built. No company updates their system every year. A few minutes seems correct for one place I worked. This was back in the 90's, before UPS technology got really interesting. Our system was two large rooms with racks and racks and racks of car batteries wired together. When the power went out, the batteries took over until the diesel generator could come online. I saw it…

Was a generator technician before I got into programming. Even the 2 megawatt systems could start up and take full load in 10-20 seconds. It sounds basically like starting your car with your foot on the gas.

The "when" shouldn't really matter- Diesel engines aren't a new thing. Warming them up isn't really a thing either- they'll have electric warmers hooked up to the building power to keep them ready to go.

Re: Tell HN: AWS appears to be down again

#612

Post by Jweb_Guru »

Hippocrates wrote: post #291
mongrelion wrote: post #205

Earlier quoted context omitted.

I agree with you. I think that having multi-AZ is the first thing to figure out before wanting to do multi-cloud, which is just another buzzword taken out of management's bullshit bucket :)

Agree, and multi AZ is usually easy. IME with AWS and GCP the control plane is the same, the scaling works across AZ, bandwidth is free and latency is near zero. The level of effort to do that is simply ticking the right boxes at setup time IME.

Cross-AZ bandwidth is far from free and the biggest reason companies avoid it (IMO). Also latency is not near zero but I don't think that's the primary reason.

Re: Tell HN: AWS appears to be down again

#613

Post by mcpherrinm »

BenjiWiebe wrote: post #540
dylan604 wrote: post #407

Earlier quoted context omitted.

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated…

Are you sure about the few minutes part? The standby generators I've seen take seconds to go from off to full load. We have an 80kw model, but I've also seen videos of load tests of much larger generators and they also take only seconds to go to full load.

I think it depends on the type of generator. I know one datacenter I worked with had turbine generators that took a few minutes to get spun up. They were started and spun up by essentially a truck engine. Those generators were quite old, though.

Re: Tell HN: AWS appears to be down again

#614

Post by fulafel »

fulafel wrote: post #533
SCdF wrote: post #305

Earlier quoted context omitted.

So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…

IME people rarely test and drill for the failovers, it's just a checkbox in a high level plan. Maybe they have a todo item for it somewhere but it never seems very important as AZ failures are usually quite rare. After ignoring the issue for a while it starts to seem risky to test for it, you might get an outage due to bugs it's likely to uncover.

Replying to myself - also in this case people are reporting that load balancing service provided by AWS failed so it doesn't necessarily help if your own stuff is tested and working.

Re: Tell HN: AWS appears to be down again

#615

Post by hinkley »

jeremyjh wrote: post #587
hinkley wrote: post #510

Earlier quoted context omitted.

Every time one of these conversations happen I end up thinking to myself that Oxide Computing needs three more competitors and a big pile of money. AWS maintains a fiction of turnkey infrastructure, and the reality of building your own is so starkly different that I haven't seen an IT group for some time that could successfully push back on these sorts of discussions. Building your own datacenter is still too much li…

I don't think putting the hardware into the rack is really the sticking point; what people like about the cloud is that it abstracts all kinds of details away for them and provides a cohesive system to manage it. AWS, Azure and Google are actually selling something like what you are talking about now [1], where for whatever legal/legacy/performance reason you need it on-prem but still want to pay AWS 5x the cost just…

That's still running their stuff in your data center.

What it tells me is that someone new needs to step in.

Re: Tell HN: AWS appears to be down again

#618

Post by fivea »

throwaway984393 wrote: post #579
fivea wrote: post #327

Earlier quoted context omitted.

> Can you expand on that? What feature do you use in east 1 that isn’t everywhere else that it’s your whole implementation? Your question reads as a strawman. It matters nothing if EC2 is also available in Mumbai or Hong Kong if by default the whole world deploys everything and anything to us-east-1, and us-east-1 alone. https://www.reddit.com/r/aws/comments/nztxa5/why_useast1_reg...

It's not a strawman. There's a huge difference between "AWS is down" and "customers don't know how to use AWS". For the people who use AWS correctly, they only had some degraded service, not downtime.

> It's not a strawman. There's a huge difference between "AWS is down" and "customers don't know how to use AWS".

Deploying a service to a single region is not, nor has it ever been, "customers don't know how to use AWS".

If anything, cargo culting this belief in global deployments being necessary, specially with services that have at most a regional demand, is a telltale sign a customer has no idea about what he is doing and is just mindlessly wasting money and engineering effort in something no one needs.

This blend of bad cargo cult advice sounds like a variant of microservices everywhere.

Re: Tell HN: AWS appears to be down again

#619

Post by dijit »

plandis wrote: post #582
dijit wrote: post #231

Earlier quoted context omitted.

What an absolutely pointless comment. Everything fails, we can argue the rate. But I would argue that understanding your constraints is better. if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place. But if you use AWS Cognito. And it goes do…

If you think you can do better than AWS, GCP, Azure there is a lot of money to be made, for sure.

Many people can do better than GCP/AWS for their workloads, but the marketing hype and fad culture around Amazon Web Services is unbeatable.

Re: Tell HN: AWS appears to be down again

#620

Post by buchanmilne »

codeduck wrote: post #606
mentat wrote: post #599

Earlier quoted context omitted.

You don't have health checks?

How are health checks supposed to help when you can't do anything?

You said:

> it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands

"Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ.

If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances.

If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way

Does that help, or have I misunderstood your problem?

Post reply on HN