AWS us-east-1 down • Hacker News Board

Re: AWS us-east-1 down

#141

Post by waz0wski » Tue, Jun 13, 2023, 8:29 PM UTC

ltcoleman wrote: post #29

We are seeing console, codebuild, etc. access issues. Possibly all using Lambdas, foundationally?

the last big us-east-1 outage was ... DNS - and it's usually DNS or software-defined core networking causing these cascading failures

Loss of DNS causes inter-service api calls to fail, then IAM and all other services fail. Anything not built to handle those situations with backoff causes a 'stampeding herd' of failure/retry and exacerbate the outage

Review the AWS statements about outages here - https://aws.amazon.com/premiumsupport/technology/pes/

Re: AWS us-east-1 down

#142

Post by JohnMakin » Tue, Jun 13, 2023, 8:29 PM UTC

I'm not sure it's the case here, but the issue with these cloud providers is they use their own services to maintain their infrastructure - that's why when something like lambda gets degraded, which would not shock me if they're using everywhere, you start to see random crap like console and IAM go down as well.

Re: AWS us-east-1 down

#143

Post by johnnyApplePRNG » Tue, Jun 13, 2023, 8:29 PM UTC

I can't even update my billing information right now :/

Re: AWS us-east-1 down

#144

Post by yawaramin » Tue, Jun 13, 2023, 8:30 PM UTC

throw0101c wrote: post #70

kroltan wrote: post #60

Earlier quoted context omitted.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

Basically just use the output of `date -u`.

Use `date -u -Iseconds`, please ;-)

Re: AWS us-east-1 down

#145

Post by Rebelgecko » Tue, Jun 13, 2023, 8:30 PM UTC

jjtheblunt wrote: post #103

TremendousJudge wrote: post #68

Earlier quoted context omitted.

It's also usually extremely US-centric. Nobody outside of North America has any idea what "PDT" or "Mountain Time" means.

And if you've been American since birth, and live in Arizona, one might still not know, since PDT and Mountain Time alternate covering Arizona seasonally. ("Ask me how i know.")

It can also varies within Arizona... one of the most confusing times in my life was driving from California through the Navajo Reservation in AZ on my way to an appointment. Was my cell phone giving me the local time on the reservation? Was it connecting to a cell tower just outside the reservation, giving me DST-less Arizona time? Or a tower slightly further away in Utah (DST?) Or was it giving me the time on the Hopi reservation, which is an enclave totally surrounded by the Navajo Reservation which uses AZ time?

Re: AWS us-east-1 down

#146

Post by paulddraper » Tue, Jun 13, 2023, 8:32 PM UTC

takeda wrote: post #39

intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

I thought it uses your browser time zone, is it not?

No. It's all PDT.

Re: AWS us-east-1 down

#147

Post by [deleted] » Tue, Jun 13, 2023, 8:33 PM UTC

[deleted]

Re: AWS us-east-1 down

#148

Post by [deleted] » Tue, Jun 13, 2023, 8:33 PM UTC

rurp wrote: post #47

intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

The inconsistency with timezones across different services in the AWS console has always baffled and annoyed me. Some places have a time without a timezone and I can never tell right away if it's utc, local time, or region time.

[deleted]

Re: AWS us-east-1 down

#149

Post by paulddraper » Tue, Jun 13, 2023, 8:34 PM UTC

dijit wrote: post #4

I wonder if this is a coincidence or if us-east-1 is simply down enough that I'm just experiencing selection bias; but I posted a poll on twitter earlier today: https://twitter.com/dijit/status/1668678588713824257 Contents: > Has anyone ever actually had customers accept an outage because AWS was down; or is this just cloud evangelicalism copium? > [ ] Yeah, outages free pass > [ ] No, they say to use AZ's

Depends on your customers.

If your customers are tech, they're too busy running around with their hair on fire too.

Re: AWS us-east-1 down

#150

Post by danryan » Tue, Jun 13, 2023, 8:34 PM UTC

nathants wrote: post #63

finally an opportunity to test a full deploy from scratch, and restore from backup, in a new region. i wonder if it will work first try? the true test of devops culture.

Good luck! My own attempt failed because SSO is down.