Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

571–580 of 646 posts

Re: Tell HN: AWS appears to be down again

#571

Post by aflag »

Spivak wrote: post #556
paulryanrogers wrote: post #177

Earlier quoted context omitted.

Spread the risk? Smaller on prem and cloud / rented bare metal?

Nah, it's actually better to concentrate the risk in this case. If your app depends on a few 3rd party services -- SendGrid, Twilio, Okta and they're all hosted on different infra then congrats! You're gonna have issues when any one of them are down, yayyy. Also the marketing benefit can't be downplayed. If your postmortem is "AWS was having issues" then your execs and customers just accept that as the cost of doing…

> Also the marketing benefit can't be downplayed. If your postmortem is "AWS was having issues" then your execs and customers just accept that as the cost of doing business because there's a built-in assumption that AWS, Azure, GCP are world class and any in-house team couldn't do it better.

In my experience, execs and customers don't treat an outage differently because AWS is at fault. Though the developers do often have the attitude that it's "someone else's problem", which can actually can make execs more worried than if the problem was well known and under the company's control.

Re: Tell HN: AWS appears to be down again

#572

Post by smiths1999 »

FpUser wrote: post #508
smiths1999 wrote: post #466

Earlier quoted context omitted.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

>"Maybe they are getting tired of arrogant older programmers..." And this is of course valid reason to ignore basic data preservation approaches. Myself I am an old fart and I realize that I am too independent / cautious. But I see way too many young programmers who just read sales pitch and honestly believe that once data is on Amazon/Azure/Google it is automatically safe, their apps are automatically scalable, etc.…

Yes - the point of that line was to be ridiculous. Age has nothing to do with it. Anyone at any age can have good ideas and bad ideas. There are some really incredibly _older_ and highly experienced engineers out there. But there are others that think that experience means they are never wrong. Age has nothing to do with this - what is important is your past experience, your understanding of the problem and then context of the problem, and how you work with your team.

And again, my point isn't that you never need backups. My point is that it is entirely plausible that at that point in time backups from S3 weren't a priority.

Re: Tell HN: AWS appears to be down again

#573

Post by smiths1999 »

lostcolony wrote: post #479
smiths1999 wrote: post #466

Earlier quoted context omitted.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

But that's just it; you can't even have that discussion if the response to "hey, should we be backing up beyond S3 redundancy?" is "No. Why would we? S3 is infallible"

Sure you can. As the experienced engineer in that setting it is a great opportunity to teach the less experienced engineers. For example, "I have seen data loss on S3 at my last job. If X, Y, or Z happen then we will lose data. Is this data we can lose? And actually, it is pretty easy to replicate - I think we could get this done in a day or two."

It's also possible the response was "That's an excellent point! I think we should put that on the backlog. Since this data is already a backup of our DB data, I think we should focus on getting the feature out rather than replicating to GCP."

Those are two plausible conversations. Instead, what we have is "these arrogant 25 year olds that have 1-2 years of experience and know it all." That's a red flag to me.

Re: Tell HN: AWS appears to be down again

#574

Post by smiths1999 »

Beltiras wrote: post #480
smiths1999 wrote: post #466

Earlier quoted context omitted.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

Losing data usually means losing customers. Usually more customers than just whos data you lost.

I suppose the caveat is you have to have customers to lose them :) We don't know what the data is or the size of the company.

Re: Tell HN: AWS appears to be down again

#575

Post by smiths1999 »

wly_cdgr wrote: post #519
smiths1999 wrote: post #466

Earlier quoted context omitted.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

Would you put the one and only copy of your family photo album up on AWS, where AWS going down would mean losing it? Because your customers' data is more important than that

AWS going down means I've lost it or temporarily lost access to it? Those are two very different scenarios. Of course S3 could lose data - a quick Google search shows it has happened to at least one account. My guess is it is rare enough that it seems like a reasonable decision to not prioritize backing up your S3 data. I'm not syaing "never ever backup S3 data" only that it seems reasonable to argue it's not the most important thing our team should be working on at this moment.

I have my family photos on a RAIDed NAS. It took me years to get that setup simply because there were higher priority things in my life. I never once thought "ahh I don't need backups of our data" I just had more important things to do.

Re: Tell HN: AWS appears to be down again

#576

Post by alostpuppy »

dragonwriter wrote: post #564
alostpuppy wrote: post #561

Earlier quoted context omitted.

Why do folks host their stuff in us-East? Is there a draw other than organizational momentum?

> Why do folks host their stuff in us-East? Off the top of my head, US-EAST-1 is: (1) topologically closer to certain customers than other regions (this applies to all regions for different customers), (2) consistently in the first set of regions to get new features, (3) usually in the lowest price tier for features whose pricing varies by region, (4) where certain global (notionally region agnostic) services are eff…

Thank you! This one is why I love HN.

Re: Tell HN: AWS appears to be down again

#578

Post by deanCommie »

RONROC wrote: post #265
f6v wrote: post #202

Earlier quoted context omitted.

Self-managed infrastructure doesn’t fail now?

We’re going to be having this same tired, pedantic, round-about conversation when Tesla’s routinely decide to take out a family of four because it mistook a plastic bag for an off-ramp. Commenters will show up like clockwork and say shit like: “What man, it’s not like cars didn’t crash before? Haha” Don’t be dense dude. And definitely don’t pursue a leadership position anytime in the future.

Tesla fans are annoying, but it is absolutely valid that the safety bar for self-driving cars can't be "100% perfectly safe" - it needs to be "safer than the alternative".

The problem with both this example, and the AWS one (it needs to have better availability than your personal home-spun solution, and it does), is that people are amazing at deluding themselves.

"Yes, cars are dangerous, because other people can't drive. But I'm a better than average driver"

"Yes, other people will build unreliable systems. But I know how to architect for my use case and ensure that for my needs the availability will be higher than AWS's"

Both are true* in the micro sense and false in the macro sense.

* Not really. 88% of americans think they are "above average" drivers.

Re: Tell HN: AWS appears to be down again

#579

Post by throwaway984393 »

fivea wrote: post #327
flatiron wrote: post #120

Earlier quoted context omitted.

Can you expand on that? What feature do you use in east 1 that isn’t everywhere else that it’s your whole implementation?

> Can you expand on that? What feature do you use in east 1 that isn’t everywhere else that it’s your whole implementation? Your question reads as a strawman. It matters nothing if EC2 is also available in Mumbai or Hong Kong if by default the whole world deploys everything and anything to us-east-1, and us-east-1 alone. https://www.reddit.com/r/aws/comments/nztxa5/why_useast1_reg...

It's not a strawman. There's a huge difference between "AWS is down" and "customers don't know how to use AWS". For the people who use AWS correctly, they only had some degraded service, not downtime.

Re: Tell HN: AWS appears to be down again

#580

Post by plandis »

TacticalCoder wrote: post #164
IceWreck wrote: post #50

Honestly my server at home has more uptime than US-East-1

I should blog about this one day but... I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this. 3 162 days. That's 8 years+ of uptime. Does it have the traffic of Amazon? No. Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013). It only has one port opened though, SSH. And with quite a…

Your server could just be an outlier. Doesn’t really say anything about AWS or any cloud provider.
Post reply on HN