Tell HN: AWS appears to be down again • Hacker News Board

Re: Tell HN: AWS appears to be down again

#481

Post by [deleted] » Wed, Dec 22, 2021, 4:50 PM UTC

laumars wrote: post #459

Nextgrid wrote: post #426

Earlier quoted context omitted.

Old-school hosting providers have a lot less moving parts than cloud providers. They have their outages, but they’re usually less frequent.

Are they though? Let’s look at what the recent AWS outages have been: a single region (but AWS makes multi-region easy). The biggest impact to most people is the AWS console, something that one seldom actually needs given AWS is API driven. If the same type of outage happened on OVH then you’d lose KVM to your physical servers. But you seldom need those either. The Azure outage was just AD service but you can roll yo…

[deleted]

Re: Tell HN: AWS appears to be down again

#482

Post by sidpatil » Wed, Dec 22, 2021, 4:51 PM UTC

aeonflux wrote: post #311

kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

> I asked what if we delete data from app by mistake? They told me we need to be careful not to do that.

Ah, the Vinnie Boombatz treatment.

Re: Tell HN: AWS appears to be down again

#483

Post by oblio » Wed, Dec 22, 2021, 4:52 PM UTC

smiths1999 wrote: post #466

aeonflux wrote: post #311

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

It's not a lot of complexity.

Add object versioning for your bucket (1 click) and mirror/sync your bucket to another bucket (a few more clicks).

Yes, your S3 costs will double, but usually they're peanuts compared to all the other costs, anyway.

Debating it takes longer than configuring it.

Re: Tell HN: AWS appears to be down again

#484

Post by ramraj07 » Wed, Dec 22, 2021, 4:52 PM UTC

smiths1999 wrote: post #466

aeonflux wrote: post #311

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

You really chose to die on “backups are for old people” as a hill?

Re: Tell HN: AWS appears to be down again

#485

Post by JackFr » Wed, Dec 22, 2021, 4:53 PM UTC

RA_Fisher wrote: post #448

ncallaway wrote: post #372

Earlier quoted context omitted.

> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…

True with two independent servers at 90% each, that’s 0.1^2 = 1% chance both fail— so redundancy can add a lot of reliability.

You assume failures are uncorrelated. Which, depending on what you think you are protecting yourself from, might or might not be true.

(Consider a buggy software release which incorrectly deletes a backup. Depending on the bug it’s very possible it will delete in both places.)

Re: Tell HN: AWS appears to be down again

#486

Post by hinkley » Wed, Dec 22, 2021, 4:55 PM UTC

kalleth wrote: post #277

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

AWS has had at least one documented incident where a region had an S3 failure that was not recoverable. They lost about 2% of all data. That might not sound like much but if you have a lot of data, partial restoration of that data doesn't necessarily leave your system in a functional state. If it loses my compiled CSS files I might be able to redeploy my app to fix it. Then again if I'm a SaaS company and that file was generated in part from user input, it might be more difficult to reconstruct that data.

Re: Tell HN: AWS appears to be down again

#487

Post by f6v » Wed, Dec 22, 2021, 4:56 PM UTC

dijit wrote: post #231

f6v wrote: post #202

Earlier quoted context omitted.

Self-managed infrastructure doesn’t fail now?

What an absolutely pointless comment. Everything fails, we can argue the rate. But I would argue that understanding your constraints is better. if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place. But if you use AWS Cognito. And it goes do…

It’s pointless to discuss how crappy cloud is whenever AWS goes down. Most of the businesses relying by the automatic RDS backups or EC2 auto scaling just don’t have time to think about all the underlying tech. I mean, I don’t manually allocate memory for variables anymore either. Do I get screwed when there’s a memory leak? Yes. What do I do about it? Move on.

Re: Tell HN: AWS appears to be down again

#488

Post by locallost » Wed, Dec 22, 2021, 4:56 PM UTC

xwdv wrote: post #453

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

You’re still in the wrong, don’t be so smug. These few downtimes are no big deal in the grand scheme of things, and your proposed solution would have been more work and headaches for little to no realizable gains, and not to mention the cybersecurity ramifications. Quite frankly, they are probably glad that you’re gone and not around to gloat about every trivial bit of downtime.

They're not gloating and also not smug. There's not even a 'hehe' in the post.

Re: Tell HN: AWS appears to be down again

#489

Post by nightpool » Wed, Dec 22, 2021, 4:57 PM UTC

oblio wrote: post #483

smiths1999 wrote: post #466

Earlier quoted context omitted.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

It's not a lot of complexity. Add object versioning for your bucket (1 click) and mirror/sync your bucket to another bucket (a few more clicks). Yes, your S3 costs will double, but usually they're peanuts compared to all the other costs, anyway. Debating it takes longer than configuring it.

As I understand it, Aeonflux was talking about redundant backups outside of S3, which are much more complex.

Re: Tell HN: AWS appears to be down again

#490

Post by bcrosby95 » Wed, Dec 22, 2021, 4:58 PM UTC

laumars wrote: post #459

Nextgrid wrote: post #426

Earlier quoted context omitted.

Old-school hosting providers have a lot less moving parts than cloud providers. They have their outages, but they’re usually less frequent.

Are they though? Let’s look at what the recent AWS outages have been: a single region (but AWS makes multi-region easy). The biggest impact to most people is the AWS console, something that one seldom actually needs given AWS is API driven. If the same type of outage happened on OVH then you’d lose KVM to your physical servers. But you seldom need those either. The Azure outage was just AD service but you can roll yo…

Most people using AWS aren't using multi-region, as evidenced by the wide array of problems on the internet when a region goes down.

I would also argue many aren't even using multiple availability zones, as evidenced by the wide array of problems on the internet when a single AZ goes down.

I think you're vastly over-estimating how most companies are using AWS, and are substituting your own requirements for theirs.

Which is very common in tech. It's part of why people shit on cloud, microservices, and other techniques large mega-corps use on HN. People write posts with lots of assumptions and few details, then people that don't know any better just carbon copy it because hey its what Google does. Meanwhile their lambda microservice system serving a blazing 60 requests per minute has more downtime than if I just hosted it on my laptop with my dialup internet connection.