Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

401–410 of 646 posts

Re: Tell HN: AWS appears to be down again

#401

Post by phil21 »

laumars wrote: post #330
Nextgrid wrote: post #285

Earlier quoted context omitted.

> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasion…

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done.

If you are going to the level of the above, you go with co-location in purpose built centers at a wholesale level. The "layer1" is all done to the specs you state and you don't have to worry about it.

On-prem rarely actually means physically on-prem at any scale beyond a small IT office room. It means co-locating in purpose built datacenters.

I'm sure examples exist, but the days of large corporate datacenters are pretty much long over - just inertia keeping the old ones going before they move to somewhere like Equinix or DRT. With the wholesalers you can basically design things to spec, and they build out 10ksqft 2MW critical load room for you a few months later.

A few organizations will find it worthwhile to continue to build at this scale (e.g. Visa, the government) but it's exceptionally small.

Re: Tell HN: AWS appears to be down again

#402

Post by numbsafari »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup.

What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experience. Still isn't fully resolved.

What happens if an admin account is taken over and your account gets screwed up?

What happens if an admin loses his shit and blows up your account?

What happens if your software has a bug that destroys a bunch of your data or fubars your account?

There's a ton of cases where having at least a simple replica of your S3 buckets into a third-party cloud could prove highly valuable.

Re: Tell HN: AWS appears to be down again

#403

Post by lmilcin »

ocdtrekkie wrote: post #370
lmilcin wrote: post #275

Earlier quoted context omitted.

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

My on prem infrastructure goes down drastically less than Amazon's. ...My home Internet even is scoring better than Amazon right now, in fact. Yours probably is too.

I have a bolt lying on my desk.

It hasn't failed since 1970 when it was produced.

It must have been built better than Space Shuttle, then.

Re: Tell HN: AWS appears to be down again

#404

Post by vel0city »

lordnacho wrote: post #317
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

What about a UPS/battery thingy? That's saved me a few times, though it normally just gives enough time for a short outage. Is it uncommon in cloud infra?

For even regular datacenters they'll often have UPS systems the size of a small car, usually several of these, to power the entire datacenters for a few minutes to get the diesel generator started.

Re: Tell HN: AWS appears to be down again

#406

Post by numbsafari »

davewritescode wrote: post #395
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

You’re not wrong but there’s ways to do backups properly in AWS and I’m not aware of there ever being an incident where AWS has lost data. It’s not a bad idea store backups offline but costs might make that an expensive proposition.

S3 isn't perfect. Read the fine print.

I've had buckets and objects disappear into the ether.

It is exceedingly rare, but it's not impossible.

Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.

Re: Tell HN: AWS appears to be down again

#407

Post by dylan604 »

tyingq wrote: post #264
vinay_ys wrote: post #217

Earlier quoted context omitted.

This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [ https://aws.amazon.com/compliance/uptimeinstitute/ ]. I really hope they publish a thorough RCA for this incident.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage.

I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated which one was hot). A major power outage hit, and there was a thunderous roar as all of the generators fired up. The channel never went off the air.

Re: Tell HN: AWS appears to be down again

#408

Post by acdha »

tluyben2 wrote: post #361
JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

> If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen.

Here are some examples where that happened:

1. Drive manufacturer had a hardware issue affecting a certain production batch, causing failures pretty reliably after a certain number of power-on hours. A friend learned the hard way that his request to have mixed drives in his RAID array wasn’t followed.

2. AC issues showed a problem with airflow, causing one row to get enough warmer that faults were happening faster than the RAID rebuild time.

3. UPS took out a couple racks by cycling power off and on repeatedly until the hardware failed.

No, these aren’t common but they were very hard to recover from because even if some of the drives were usable you couldn’t trust them. One interesting dynamic of the public clouds are that you tend to have better bounds on the maximum outage duration, which is an interesting trade off compared to several incidents I’ve seen where the downtime stretched into weeks due to replacement delays or manual rebuild processes.

Re: Tell HN: AWS appears to be down again

#409

Post by vel0city »

notyourday wrote: post #241
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes…

The only full datacenter outage I've personally experienced was a power maintenance tech testing the transfer switch between systems where the power was 90 degrees out of phase. Big oof.

Re: Tell HN: AWS appears to be down again

#410

Post by fatnoah »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS.

At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, even when including the additional person-hours required to maintain it on our end.

Keeping our infra simple helped that work, and it's literally been years since an outage caused by any AWS issues, even though there have been several large AWS events.

Post reply on HN