Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 down

news.ycombinator.com

Post reply on HN
Print view View original

301–310 of 331 posts

Re: AWS us-east-1 down

#301

Post by great_psy »

nfRfqX5n wrote: post #263

> At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services. As of 3:37 PM, the backlog was fully processed pretty cool that stuff was stored in a backlog and eventually processed!

I am guessing they mean SQS queue … so basically the SQS aid doing what is supposed to: - try to process event - send to DLQ if it fails - there’s a redrive button in the DLQ to … well restive the events after the lambda is fixed

Re: AWS us-east-1 down

#302

Post by smohare »

kroltan wrote: post #60
intsunny wrote: post #28

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

[dead]

Re: AWS us-east-1 down

#303

Post by KingMachiavelli »

Weird, I didn't notice the actual outage at all except high ping to a non-AWS server/IP on the west coast. Normally the latency is ~85ms, today it has been >170ms. SSH is basically unusable at that latency and even bandwidth is very low (not sure how that could be).

Re: AWS us-east-1 down

#304

Post by jamesfinlayson »

xp84 wrote: post #196
rurp wrote: post #47

Earlier quoted context omitted.

The inconsistency with timezones across different services in the AWS console has always baffled and annoyed me. Some places have a time without a timezone and I can never tell right away if it's utc, local time, or region time.

> The inconsistency [of everything, everywhere] in the AWS console ftfy AWS is powerful and very popular, but for the console, "it functions" must be the only condition the UI has to satisfy. Should every page use a unique table and sorting widget and UI language? Yes, please! I'm assuming this helps them move fast, not having to coordinate with anybody or wait for a UI designer to tell them how it should look. But i…

I've been told that each service is responsible for their own UI.

Re: AWS us-east-1 down

#305

Post by jamesfinlayson »

heleninboodler wrote: post #258
ishjoh wrote: post #111

Earlier quoted context omitted.

certificate manager also down (I know because I tried to update an ssl cert for cloudfront which only allows US-East-1 ssl certs, maybe someone will eventually fix that to allow any region to have the ssl cert for cloudfront)

> cloudfront which only allows US-East-1 ssl certs This seems like an odd limitation. Do you know the technical reason?

CloudFront is a global service according to AWS (I believe you pay more if you want your content in CDNs in more/different regions' edges).

Re: AWS us-east-1 down

#306

Post by jlawer »

inopinatus wrote: post #273
sk0g wrote: post #267

Earlier quoted context omitted.

Even in Australia, AEST has a DST flavour and a non-DST one. Queensland does not observe DST while the other states do. You can drive around a roundabout at the border and switch timezones for fun. Or go down there to celebrate the new year twice. Or go from rabbits being OK to some 5 figure fine if you're caught with one :)

my limited understanding of local pejoratives suggests that it remains traditional to sledge QLD for being one hour and twenty years behind

I believe its traditional to reply that NSW was 8 points behind in the only yard stick those north of the border care about.

https://en.wikipedia.org/wiki/State_of_Origin_series

Re: AWS us-east-1 down

#307

Post by hosteur »

YetAnotherNick wrote: post #287
hosteur wrote: post #250

Earlier quoted context omitted.

If people do care they could use other hosting providers such as Hetzner or OVH, no?

Why did you come to the conclusion that Hetzner or OVH is more reliable. At least their SLA credits doesn't say that.

I don’t think any individual provider would be significantly more reliable. But it would make the landscape more diverse if people did not put all their eggs in the big three providers.

Re: AWS us-east-1 down

#308

Post by otterley »

paulddraper wrote: post #176
dijit wrote: post #46

Earlier quoted context omitted.

It's the: * Largest (DDoS'd most, most complex, scaling issues etc) * Oldest (More time for weird idiosyncrasies to take hold) * Where most testing happens * Where new products are deployed first

It's also the home of single region services... IAM, Cloudfront ACM certs, etc

Those are not single-region services. Changes must be executed there, but the data is replicated globally. If you don’t need to make changes in the context of those services, they will keep working in the other regions even during an incident in the primary region.

Re: AWS us-east-1 down

#309

Post by otterley »

maxcan wrote: post #83
impulser_ wrote: post #69

Why does everyone keep deploying their products to this one region when it always seems like the one that fails? We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?

There are some services (cloudfront for example) which require this region. Its not that much harder to have multiple regions in your deployment but putting everything in one is simpler for smaller startupy orgs.

It’s not required during an incident. The data for these services is globally replicated. It’s only if you need to make changes that you might be impacted if you’re already successfully operating out of another region.

Re: AWS us-east-1 down

#310

Post by otterley »

paulddraper wrote: post #230
impulser_ wrote: post #69

Why does everyone keep deploying their products to this one region when it always seems like the one that fails? We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?

Features. For example, do you want your Cloudfront CDN to have a custom (secure) domain? Then you have to host your ACM cert in us-east-1.

If you’ve already set up your ACM cert, you won’t be impacted during an incident in us-east-1. It’s only if you need to make changes that you could be impacted.
Post reply on HN