Live data from Hacker News

AWS us-east-1 down

news.ycombinator.com

311–320 of 331 posts

Re: AWS us-east-1 down

#312

Earlier quoted context omitted.

That sounds nice. My boss's boss is usually the one storming in, and he usually says "okay let's do it", and then I have to implement it in a week...

That’s why you always BofH the estimates to include some fun toys for yourself, too

[flagged]

Re: AWS us-east-1 down

#313

It doesn't matter if your infra is in another region, because there will almost always be transitive dependencies on us-east-1. IAM is deployed in us-east-1 and there will always be a transitive dependency on us-east-1

So much for redundancy I guess.

Re: AWS us-east-1 down

#315

Earlier quoted context omitted.

just report it in epoch seconds

better: time encoded as a float of trecenti-seconds since year -8435 of the Georgian Calendar why? 'caus it hurt to even just think about implementing that anywhere

Ignoring leap seconds?

Re: AWS us-east-1 down

#317
post #238

Earlier quoted context omitted.

How does this generate off-by-one errors? I am also part of a company with employees in pretty much every timezone, but when they create a meeting the meeting invitation is programmed with the correct timezone so in my Calendar it always shows what time the meeting is going to be for me. I never even have to think what timezone the organizer is...

The off-by-one error occurs when you announce an event in Standard time but really mean Daylight time, or vice versa. While those local to the time zone will often automatically correct this mistake either consciously on unconsciously, those in other time zones (especially where Daylight time isn't used or is on a different schedule) will tend to rely on time conversion tools which will take a literal interpretation…

The fact that you have to announce timezones is already an error. If I need to schedule a meeting I don't need to select timezones, they're already selected from the timezone I'm part of. There's never room for error by "picking" the wrong thing, since there's nothing to pick. And if my system is programmed with the wrong timezone, then every single meeting will be off-by-N and my calendar will show the wrong time as "now". It would be impossible to miss such an error.

I think your company needs better tools to handle meetings.

Re: AWS us-east-1 down

#318
post #60

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

> Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

Many people also get the timezone names completely wrong. I've had multiple scheduling email exchanges where someone says X pm EST not realizing that at the time it's currently EDT and that EST ≠ EDT.

And yet, for some reason, the two-letter abbreviations (e.g., ET) that are technically correct year-round, never seem to have caught on in the wild.

I've given up on the abbreviations and just say "Eastern" now to avoid confusion.

Re: AWS us-east-1 down

#319
post #236
post #183

Earlier quoted context omitted.

True in theory, in practice people often get it wrong and use the incorrect one.

Indeed. There are Americans who will tell me PST, when they meant PDT but forgot to mention that. Now I have to track the American DST calendar as well as European DST calendar to do the conversion. There are also people who tell me GMT (because they think that term means "the time in London") when they meant BST (because in summer, London doesn't operate on GMT).

Most Americans, at least non-engineers, will incorrectly say PST, EST, etc year-round when they actually mean PT, ET, etc.

I have found this site very helpful for linking people to when they are confused or using them incorrectly:

https://time.is/PT

https://time.is/PST

https://time.is/PDT

The time zone comparison feature is nice as well:

https://time.is/compare/0800AM_14_June_2023_in_Cincinnati/Lo...

Re: AWS us-east-1 down

#320
post #204

It doesn't matter if your infra is in another region, because there will almost always be transitive dependencies on us-east-1. IAM is deployed in us-east-1 and there will always be a transitive dependency on us-east-1

I have never had a production issue in other regions due to a us-east-1 outage. The worst that ever happened was I had to wait to update a Cloudfront distribution because the control plane (based in us-east-1) was down, but the existing configuration continued working fine throughout. I don't know what the architecture of IAM looks like, but somehow it's never suffered a global outage. AWS is really, really good at r…

I have.

Not being able to update your existing resources is still an outage from a DevOps perspective.

It might be an API level outage vs an end-user level outage from your customer's perspective, but if the functionality is down, it's an outage.

Post reply on HN