Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 down

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 331 posts

Re: AWS us-east-1 down

#61

Post by robrtsql »

fbellag wrote: post #33
robrtsql wrote: post #10

Earlier quoted context omitted.

> No, they say to use AZ's Using 3 AZs in us-east-1 won't save you. I guess a demanding customer would have said 'you should have implemented disaster recovery so you could failover to us-east-2' but that's easier said than done. The more regional AWS services you adopt, the bigger the impact is. How does one recover from a regional outage if their pipeline is in that region?

What I did once I was in the position of _having_ to provide that level of support, was to run the pipeline in a third region, different from the "prod" ones. That way, worst case you can't do deployments during the outage... Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run

> It was discarded bc you never know where that would actually run

Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.

Re: AWS us-east-1 down

#62

Post by throwaway019254 »

nicoffeine wrote: post #40
whoisjuan wrote: post #34

Why is it always us-east-1 though? I have always stayed away from that region because it seems significantly less reliable than other regions.

I thought I read that this is where they deploy new changes first. Can anyone confirm?

I don't believe it's true. I was working on one of the biggest AWS services and we always deployed to small regions first.

@dijit is right: https://news.ycombinator.com/item?id=36315736

Re: AWS us-east-1 down

#63

Post by nathants »

finally an opportunity to test a full deploy from scratch, and restore from backup, in a new region.

i wonder if it will work first try? the true test of devops culture.

Re: AWS us-east-1 down

#64

Post by Johnny555 »

dijit wrote: post #4

I wonder if this is a coincidence or if us-east-1 is simply down enough that I'm just experiencing selection bias; but I posted a poll on twitter earlier today: https://twitter.com/dijit/status/1668678588713824257 Contents: > Has anyone ever actually had customers accept an outage because AWS was down; or is this just cloud evangelicalism copium? > [ ] Yeah, outages free pass > [ ] No, they say to use AZ's

My employer lets customers choose which of our supported regions to run in and exempts cloud provider outages from our SLA (we’re on the hook for staying up for single AZ outages, but not multi AZ or region outages). We provide tools to help customers replicate their data so they can be multi-region or even multi provider if they want to.

Re: AWS us-east-1 down

#65

Post by colanderman »

kroltan wrote: post #60
intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

And even if the time is UTC, please indicate this.

Re: AWS us-east-1 down

#66

Post by dylan604 »

kroltan wrote: post #60
intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

just report it in epoch seconds

Re: AWS us-east-1 down

#68

Post by TremendousJudge »

kroltan wrote: post #60
intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

It's also usually extremely US-centric. Nobody outside of North America has any idea what "PDT" or "Mountain Time" means.

Re: AWS us-east-1 down

#69

Post by impulser_ »

Why does everyone keep deploying their products to this one region when it always seems like the one that fails?

We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?

Re: AWS us-east-1 down

#70

Post by throw0101c »

kroltan wrote: post #60
intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

Basically just use the output of `date -u`.
Post reply on HN