Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 646 posts

Re: Tell HN: AWS appears to be down again

#241

Post by notyourday »

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time.

This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale.

Data centers lose supply lines. They lose transformers. Sometimes they lose primary feed and secondary feed at the same time. Automatic transfer switches cannot be tested periodically i.e. they are typically tested once. Testing them is not "fire up a generator and see if we can draw from it"

It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations.

Re: Tell HN: AWS appears to be down again

#242

Post by exikyut »

colinbartlett wrote: post #157
hnarn wrote: post #14

Is there a history of AWS downtimes available somewhere? This makes what, three times in as many months? edit: The question isn't necessarily AWS specific, just any data on amount of downtime per cloud provider on a timeline would be nice.

I have tons of this kind of data due to my side project, StatusGator. For some services like the big cloud providers I have data going back 7 years. There indeed has been an uptick in AWS outages recently. You can see a bit of the history here: https://statusgator.com/services/amazon-web-services

(I was idly curious. It appears this data is available as part of the ~US$280/mo tier, along with a bunch of other things.)

Re: Tell HN: AWS appears to be down again

#243

Post by ItsBob »

uluyol wrote: post #233
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

Why spend the cost on dual X and Y when you can failover to another cluster? For big DC workloads, it is usually, though not always, better to take the higher failure rate than add redundancy.

Really? You'd think at Amazon's scale an additional PSU in a 1U custom-built server (I assume they're custom) would be a few tens of $ at most.

Actually, now that I type that it makes sense. Scaling a few tens of dollars to a bajillion servers on the off-chance that you get an inbound power failure (quite rare I'd reckon) might cost more than what they'd lose if it does actually fail.

So yeah, they're potentially just balancing the risk here and minimising cost on the hardware.

Edit: changed grammar a bit.

Re: Tell HN: AWS appears to be down again

#244

Post by [deleted] »

ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

[deleted]

Re: Tell HN: AWS appears to be down again

#245

Post by gilrain »

junon wrote: post #223
gilrain wrote: post #213

Earlier quoted context omitted.

You make more friends defending humans from big companies than you do defending big companies from humans. Your argument would have a small amount of merit if you acknowledged that the laws DO NOT protect people like they do corporations. That is a hollow ideal, not reality.

I don't really see your point. My original comment was more pointing out that the operators, if not Amazon, could be seen as infringing their rights. They should update their legalese if they want to be truly protected. How is that defending Amazon? Regardless of your pointed comment, I'm operating in the land of legal objectivity. The law doesn't care about your feelings much.

Your argument is that IP law should be equally observed by everyone because it protects individuals and corporations alike:

> Nope. Because these laws also protect people who make such websites from the corporations they're commenting on, too.

My response is that your assumption is very obviously wrong: the law does not protect individuals and corporations alike.

That’s all.

Re: Tell HN: AWS appears to be down again

#246

Post by FridayoLeary »

AQuantized wrote: post #149
alecbz wrote: post #138

Earlier quoted context omitted.

I wonder to what extent this actually becomes less of a problem the more people use AWS. At this point AWS being down just feels like "the internet is down", it's hard for customers to be too mad at any company being down when all their competitors are too. Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.

This seems like an interesting pendulum swing where the few companies not reliant on AWS could capture significant enough revenue by maintaining uptime during a potential busy season outage.

Maybe they'll just find themselves DDOSed by the sudden influx of visitors? As a small example, i think HN was slightly slower when FB had their outage.

Re: Tell HN: AWS appears to be down again

#247

Post by seeking_future »

2-718-281-828 wrote: post #225
itsoktocry wrote: post #216

Earlier quoted context omitted.

> is that just playing with words? It conveys reality, that "fail-safe" isn't literal, as if anyone believed that.

I mean it has to be play with words or tongue in cheek simply b/c the assumption of a fail-safe system failing is already contradictory. So you cannot say anything smart about that beyond - there are no fail-safe systems that fail.

The real world is the play. Words are just catching up.

Re: Tell HN: AWS appears to be down again

#248

Post by doublerabbit »

ryanbrunner wrote: post #193
christophilus wrote: post #147

Earlier quoted context omitted.

Most people don't need to handle the traffic of US-East-1. They just need a single, simple, mostly reliable server. But they're often told, "Don't do that. It's too hard, and irresponsible, and what if you get a spike in traffic, and what if you need to add 5 new servers, and security is really hard." In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the wor…

I think most people on here are coming from the perspective of startups, which scale out of a single server setup pretty quickly. At a bare minimum, most will have dedicated purpose-built servers like Redis or a DB, and often there's separate background workers, or a load balancer with a couple of web servers. When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first…

> running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.

What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required.

If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challenge other than maintaining X amount of hardware than just one.

Re: Tell HN: AWS appears to be down again

#249

Post by ItsBob »

notyourday wrote: post #241
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes…

Yes but if you have reliable power from two different sources then the biggest risk (I'd imagine) is the failover circuitry! Something that should be tested tbh.

Also, there are banks of batteries and generators in between the power company cables and the kit: did they not kick-in?

Again, this is all pure speculation: I have absolutely no idea of the exact failure, nor how their infrastructure is held together - this is all just speculation for the hell of it :)

Re: Tell HN: AWS appears to be down again

#250

Post by bombcar »

notyourday wrote: post #241
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes…

The datacenter we were in had dual-sourced grid power (two separate grid connections on opposite sides of the block, coming from different substations) along with a room of batteries (good for iirc 1hr total runtime for the whole datacenter, setup in quad banks, two on each "rail"), _and_ multiple independent massive diesel generators, which they ran and switched power to every month for at least an hour.

And to top it off each rack had its own smaller UPS at the bottom and top, fed off both rails, and each server was fed from both.

We never had a power issue there; in fact SDGE would ask them to throw to the generators during potential brown-out conditions.

Of course this was a datacenter that was a former General Atomics setup iirc ...

Post reply on HN