Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 down

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 331 posts

Re: AWS us-east-1 down

#161

Post by d8dUf7KjBEYCk7Q »

matwood wrote: post #123
kobalsky wrote: post #71

Earlier quoted context omitted.

it's important to inform customers about the resiliency of their systems and let them pick how far they are going to invest for it. then you get to eat popcorn when stuff explodes. * single server event. $ * multi server event. $$ * single az event. $$$ * multi az event. $$$$ * global provider event. $$$$$ * cross provider event. $$$$$$ * alien invasion. $$$$$$$$$$$$$$

Back when we had servers in an onsite DC we lost a raid card and the system I was developing went down. We had the fancy support so a tech was out with the card replaced in a couple hours, then we had to restore from tape backup. All in all, a non-critical system was down for most of a business day. My bosses boss stormed in, upset he couldn't pull a report, and asked how do we prevent this in the future. I responded…

That sounds nice. My boss's boss is usually the one storming in, and he usually says "okay let's do it", and then I have to implement it in a week...

Re: AWS us-east-1 down

#162

Post by Travis_Pastrami »

dyingkneepad wrote: post #156
NoZebra120vClip wrote: post #105

Earlier quoted context omitted.

3 years ago, when I started work for my current employer, I noticed in Slack that everyone was reckoning time in "Standard Time" year-round. Now imagine my chagrin because I live in Arizona, and "Mountain Standard Time" does not change for DST. Therefore, all my coworkers were citing nonsensical, nonexistent time zones and it was messing up my ability to convert back and forth. Come to find out that this was some sor…

How does this generate off-by-one errors? I am also part of a company with employees in pretty much every timezone, but when they create a meeting the meeting invitation is programmed with the correct timezone so in my Calendar it always shows what time the meeting is going to be for me. I never even have to think what timezone the organizer is...

It's the same at my company. Teams and Zoom both automatically schedule meetings in every attendees' own time zone. Maybe that person's company still does phone meetings or something.

Re: AWS us-east-1 down

#163

Post by vultour »

kinghajj wrote: post #15
dijit wrote: post #4

I wonder if this is a coincidence or if us-east-1 is simply down enough that I'm just experiencing selection bias; but I posted a poll on twitter earlier today: https://twitter.com/dijit/status/1668678588713824257 Contents: > Has anyone ever actually had customers accept an outage because AWS was down; or is this just cloud evangelicalism copium? > [ ] Yeah, outages free pass > [ ] No, they say to use AZ's

AZs don't really help when it's AWS' own services across the entire region that break. Anecdotally, we have had customers accept outages that were out of our control without penalty.

AZs also don't help with natural disasters at all. I believe AWS is the only one doing geographically distributed AZs, for the others it just means different connections and placed somewhere else in the building.

edit: turns out AWS is the one with geo distribution, not Azure

Re: AWS us-east-1 down

#165

Post by TonyCoffman »

aeyes wrote: post #90
gtirloni wrote: post #59

Earlier quoted context omitted.

DCs are pretty static and offer way fewer services than AWS or any other public cloud. I worked for one for some time and whenever we had issues, some people would call and ask if we were going bankrupt. It gave me a feeling they also have way smaller customers that might not understand the underlying stack.

If all you use in AWS is static EC2 instances you would have to go back a looooong time to find an outage which affected their availability. Even in us-east-1.

December 22, 2021 was the last partial impact we had in us-east-1 for EC2 instances. They had power issues in USE1-AZ4 that took a while to sort out.

Re: AWS us-east-1 down

#166

Post by NoZebra120vClip »

Travis_Pastrami wrote: post #162
dyingkneepad wrote: post #156

Earlier quoted context omitted.

How does this generate off-by-one errors? I am also part of a company with employees in pretty much every timezone, but when they create a meeting the meeting invitation is programmed with the correct timezone so in my Calendar it always shows what time the meeting is going to be for me. I never even have to think what timezone the organizer is...

It's the same at my company. Teams and Zoom both automatically schedule meetings in every attendees' own time zone. Maybe that person's company still does phone meetings or something.

We don't use any automatic scheduling with Zoom or Google Calendar. Management doesn't send invites to those meetings, they just publish the link on Slack and we have to figure out how to get it into our calendars.

Trust me, at least once I missed a meeting because I was late by an hour due to time zone confusion.

Re: AWS us-east-1 down

#167

Post by WirelessGigabit »

NoZebra120vClip wrote: post #105
TremendousJudge wrote: post #68

Earlier quoted context omitted.

It's also usually extremely US-centric. Nobody outside of North America has any idea what "PDT" or "Mountain Time" means.

3 years ago, when I started work for my current employer, I noticed in Slack that everyone was reckoning time in "Standard Time" year-round. Now imagine my chagrin because I live in Arizona, and "Mountain Standard Time" does not change for DST. Therefore, all my coworkers were citing nonsensical, nonexistent time zones and it was messing up my ability to convert back and forth. Come to find out that this was some sor…

I just had someone asking me if I'm available at 5pm EST.

Also, your clock can get confused driving North from PHX to Zion National Park.

In summer you start in Mountain Standard Time, drive into the Navajo Nation which does observe Mountain Daylight Time, containing through the Hopi Reservation, which is Mountain Standard Time. Then you end up back in Navajo Nation with Mountain Daylight Time. You keep on driving towards Page which is in Mountain Standard Time. However, when you cross the state-border of AZ/UT you're back in Mountain Daylight Time.

My clock threw a segmentation fault.

Re: AWS us-east-1 down

#169

Post by ramranch »

ciguy wrote: post #17
dveeden2 wrote: post #13

https://health.aws.amazon.com/health/status reports: Increased Error Rates and Latencies Jun 13 12:08 PM PDT We are investigating increased error rates and latencies in the US-EAST-1 Region. They list Lambda as the only affected service

They've added a dozen or so more as potentially down now. Anything that uses IAM, which I suspect is the core of the issue.

doesn't every service use IAM?
Post reply on HN