Live data from Hacker News

AWS us-east-1 down

news.ycombinator.com

291–300 of 331 posts

Re: AWS us-east-1 down

#291

Earlier quoted context omitted.

I thought I read that this is where they deploy new changes first. Can anyone confirm?

I don't believe it's true. I was working on one of the biggest AWS services and we always deployed to small regions first. @dijit is right: https://news.ycombinator.com/item?id=36315736

I have a suspicion that AWS uses some regions as canaries. Because we control both ends of things, I have personally noted that certain AWS functions clearly break in Australia first.

Re: AWS us-east-1 down

#292

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

https://aws.amazon.com/about-aws/whats-new/2022/09/aws-healt...

If you login, you can specify what timezone for timestamps and for the text to be parsed into your timezone preference.

Re: AWS us-east-1 down

#293
post #60

Earlier quoted context omitted.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

It's also not too complicated to add a few lines of javascript that show the datet/time in the user's local time zone (via Date.getTimezoneOffset) as well.

They did that: https://aws.amazon.com/about-aws/whats-new/2022/09/aws-healt...

Re: AWS us-east-1 down

#294
post #60

Earlier quoted context omitted.

Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.

Basically just use the output of `date -u`.

https://docs.aws.amazon.com/health/latest/ug/getting-started...

Re: AWS us-east-1 down

#295
post #110

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

Technically PDT is always 7 hours behind UTC. PST is always 8 hours behind. We just change which one we use twice a year. Pacific time makes sense when you realize Fremont is the center of the universe.

Yup, this is why I always say “US ET” (I'm on the east coast). I don't trust myself or anyone else to get it right, and if the other party is converting anyway, their conversion tool (google?) should be able to handle that. (Of course, the date is necessary but implicit, but that's usually fine too.)

Re: AWS us-east-1 down

#296

Earlier quoted context omitted.

date: illegal option -- I usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]

Let me guess, you are on a Mac.

Probably on an older Mac or FreeBSD. On my macOS 12.6, `man date` says:

    The -I flag was added in FreeBSD 12.0.

Re: AWS us-east-1 down

#297

Earlier quoted context omitted.

Ok fine. Running your own datacenter in 2023 is incredibly risky. There's the upfront server cost and the ongoing maintenance cost. There's patches and staffing and disaster planning and all the other things that goes into it. Plus there's the cyberinsurance and protections and security components too. Do you really think other (smaller) orgs can do a better job at hosting a datacenter than Amazon / Google / Microsof…

> Running your own datacenter in 2023 is incredibly risky. There are middle grounds. But let's be honest: 99% of companies have never done the napkin math, because nobody ever got fired for choosing IBM^W AWS. We joked about this in my company: we had a variable-load thing that we used autoscaling in the cloud for, but it had a baseline load that purchasing a real machine might have made a lot of sense for. The napki…

Excellent response, thank you. To expand on the cybersecurity aspect - think of services like Cloudflare WAF and DDoS protection. These services are very easy for orgs to implement and do a really good job at covering 95% of threats quickly.

Could you imagine a 1,000 person org with a 20 person IT department rolling their own DDoS solution?

But yes you are also right, cyber insurance is still required, and even AWS touts an expectation of a “shared responsibility” model.

I’m still skeptical that cloud hosted offerings are a bad thing. For a long time there were only Ford, Chrysler and Chevy in America, then foreign imports became popular, then a few years ago Tesla became a contender.

I still think new entrants can come into the cloud space, particularly in Europe, but they need to do their due diligence and understand their competitors offerings very well.

Re: AWS us-east-1 down

#298
post #279

I "love" it when my vacuum stops working because an online book sellers servers went down. #modernlife This is a good reminder to avoid cloud-centric products, but they are getting harder and harder to avoid.

Did this actually happen? The vacuum part

Yes. This morning I found my Roomba (i9 something IIRC) sitting idle in the middle of the kitchen. When I launched the iOS app, it appologised it couldn't connect due to AWS being down (I regret not taking a screenshot).

Re: AWS us-east-1 down

#299

For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.

I've set the option on the below page to UTC https://health.aws.amazon.com/health/status#settings As I'm logged in, it persists across browser sessions.

It's worse.

Imagine you have two browser opening the same page, one showing UTC, another showing your local time.

There are no indicator showing which time zone is used. You have to mentally correlate "this browser windows has logged in..." with the time shown on screen.

Re: AWS us-east-1 down

#300

Earlier quoted context omitted.

That's my favorite part of Amazon's console. That miniature heart attack you have when you ask "WHERE ARE ALL MY LAMBDAS AND DYNAMO INSTANCES?" Then you realize that they just switched you back to us-east-1 for some reason and a wave of familiar relief washes over you.

Out of curiosity have you been building with the serverless framework? I'm curious what drives people to use dynamodb in particular

No, I really don't like any of the serverless or amplify type of frameworks that AWS produces.

What I needed was a distributed store for managing a large number of small configuration "files" and other state "links" across the infrastructure I built. In particular, I needed the ability for a write conflicts to be detected and managed immediately and for consistent reads to be available in some cases.

Looking at the size of data and number of transactions, DynamoDB on a per-request model was going to be significantly cheaper and easier than standing up a bunch of DB instances or other "fixed" infrastructure.

The ultimate model was to actually implement something like a POSIX filesystem on top of the "nosql" layer. In practice, I've been really happy with that decision, and it makes working on the code that interacts with this system very familiar and easy to understand. It's even got symlinks, acls, and automatically expiring advisory locks.

Post reply on HN