Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

441–450 of 646 posts

Re: Tell HN: AWS appears to be down again

#441

Post by thraxil »

numbsafari wrote: post #406
davewritescode wrote: post #395

Earlier quoted context omitted.

You’re not wrong but there’s ways to do backups properly in AWS and I’m not aware of there ever being an incident where AWS has lost data. It’s not a bad idea store backups offline but costs might make that an expensive proposition.

S3 isn't perfect. Read the fine print. I've had buckets and objects disappear into the ether. It is exceedingly rare, but it's not impossible. Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.

> Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.

With the caveat that you're going to have to implement all your access controls, monitoring and compliance mechanisms on those alternate backups. No point winning points during an audit for having backups outside AWS if you lose even more points for "backups weren't properly secured against unauthorized access".

And you're regularly restoring from those alternate backups as well to check their integrity, right?

Re: Tell HN: AWS appears to be down again

#442

Post by hinkley »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Have you contacted them to see how things are going?

Maybe a cheery note asking how the team is doing, sent right in the middle of an outage.

Passive aggressive? As hell. Cathartic? Damn skippy.

Re: Tell HN: AWS appears to be down again

#443

Post by sigstoat »

tzs wrote: post #364
laumars wrote: post #330

Earlier quoted context omitted.

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

> You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages. I don't see why the petrol station needs to be a short distance away. Unless the plan is to walk to the petrol station and back (which should not be the plan[1]), anyplace within reasonable driving distance should do. [1] long duration elec…

> I don't see why the petrol station needs to be a short distance away

some natural disasters can render driving trickier than walking. extremely large snow storms, for instance. you can still walk a block, but you might be hard pressed to drive 5 miles.

(i don't have a bone in this particular cautiousness-fight; personally i'd just suggest folks producing DR plans cover the relevant natural disasters for the area they live in, while balancing management desires, and a realistic assessment of their own willingness to come to work to execute a DR plan during a natural disaster.)

Re: Tell HN: AWS appears to be down again

#444

Post by tyingq »

dylan604 wrote: post #407
tyingq wrote: post #264

Earlier quoted context omitted.

"Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain fu…

The generators should be powering up as soon as one of the 2 different sources goes down. It takes generators a few minutes to power up and get "warmed up". If they don't start this process until both mains sources are down, then oops, there's power outage. I used to work next door to a "major" cable TV station's broadcast location. They had multiple generators on-site, and one of them was running 24/7 (they rotated…

There are setups where the UPS is designed to last long enough for generator spin up as well. I believe it's the most common setup if you have both. I assume spinning up the generators for very short-lived line power blips might be undesirable.

Re: Tell HN: AWS appears to be down again

#445

Post by pkulak »

lmilcin wrote: post #275
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

> Can you make your on prem infrastructure go down less than Amazon's?

Over the last two years, my track record has destroyed AWS. I've got a single Mac Mini with two VMs on it, plugged in to a UPS with enough power to keep it running for about three hours. It's never had a second of unplanned downtime.

About 15 years ago I got sick of maintaining my own stuff. I stopped building Linux desktops and bought an Apple laptop. I moved my email, calendars, contacts, chat, photos, etc, to Google. But lately I've swung 180 degrees and have been undoing all those decisions. It's not as much of a PITA as I remember. Maybe I'm better at it now? Or maybe it will become a PITA and I'll swing right back.

EDIT: I realize you're talking in a commercial sense and I'm talking about a homelab sense. Still, take my anecdote for what it's worth. :D

Re: Tell HN: AWS appears to be down again

#446

Post by darkstar999 »

pkulak wrote: post #425

I used to think it was silly to have your own hardware (like a NAS) in your house. What makes you think you can do it better than AWS? Santa is bringing me a Synology in three days.

Why not both? I just got a Synology NAS and it makes cloud sync dead simple. Now the most important things are on my PC, mirrored on 2 drives in my NAS, and on AWS S3 (or any other cloud storage).

Re: Tell HN: AWS appears to be down again

#447

Post by xboxnolifes »

manquer wrote: post #411
ncallaway wrote: post #372

Earlier quoted context omitted.

> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…

My point is if your redundancy is better than AWS then why pay for them ? If it not they why invest in your own?. You can argue that you protect against different threats than AWS does . So far I have not seen a meaningful argument of threats a on Prem protects differently than the cloud that you need both . Say for example your solution is to put all your data backups on the moon then it makes sense to do both, AWS…

> My point is if your redundancy is better than AWS then why pay for them ? If it not they why invest in your own?

This is a really confusing question. Redundancy requires more than 1 option. It's not about it being better than AWS, it's that in order to have it you need something besides just AWS. AWS may provide redundant drives, but they don't provide a redundant AWS. AWS can protect against many things, but it cannot protect against AWS being unavailable.

Re: Tell HN: AWS appears to be down again

#448

Post by RA_Fisher »

ncallaway wrote: post #372
manquer wrote: post #342

Earlier quoted context omitted.

S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.

> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…

True with two independent servers at 90% each, that’s 0.1^2 = 1% chance both fail— so redundancy can add a lot of reliability.

Re: Tell HN: AWS appears to be down again

#449

Post by boudin »

acdha wrote: post #394
boudin wrote: post #357

Earlier quoted context omitted.

Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well. And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything. It's AWS job to ensure that things are reliable, that t…

Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else. I do agree th…

I hope AWS will provide some explanation about those issues and what actions they will take to prevent those in the future

On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.

Re: Tell HN: AWS appears to be down again

#450

Post by frupert52 »

2-718-281-828 wrote: post #225
itsoktocry wrote: post #216

Earlier quoted context omitted.

> is that just playing with words? It conveys reality, that "fail-safe" isn't literal, as if anyone believed that.

I mean it has to be play with words or tongue in cheek simply b/c the assumption of a fail-safe system failing is already contradictory. So you cannot say anything smart about that beyond - there are no fail-safe systems that fail.

Do you mean in that it fails by failing to be the thing that it purports to be? Making it no longer that thing? At what point does bread become toast?
Post reply on HN