How many folks actually use multi-region deployments with automatic failover (e.g. latency based routing in route 53)?
AWS us-east-1 down
231–240 of 331 posts
Re: AWS us-east-1 down
#232Why does everyone keep deploying their products to this one region when it always seems like the one that fails? We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?
There's a lot of software, iirc even Amazon's own dashboards, that simply defaults to us-east-1.
FWIW, mine defaults to Ohio. Has happened multiple times. IDK if it is geographic or what.
Re: AWS us-east-1 down
#233Earlier quoted context omitted.
Just need to deploy your service on Mars AND Earth. Duh
And you thought the current time zone confusion was bad. Now you have two sets of time zones, and a varying delay of about 5 to 21 minutes between them. Oh, the joy!
note to self: synchronous replication may be a problem
Re: AWS us-east-1 down
#234For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.
Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.
And if it's on a forum debating an event that's about to happen soon, I find the following extremely convenient:
- the keynote will start when this post is 5 hours old
- the rocket launch is scheduled to when this comment is 30 hours old
Re: AWS us-east-1 down
#235Re: AWS us-east-1 down
#236Earlier quoted context omitted.
Technically PDT is always 7 hours behind UTC. PST is always 8 hours behind. We just change which one we use twice a year. Pacific time makes sense when you realize Fremont is the center of the universe.
True in theory, in practice people often get it wrong and use the incorrect one.
Indeed. There are Americans who will tell me PST, when they meant PDT but forgot to mention that. Now I have to track the American DST calendar as well as European DST calendar to do the conversion.
There are also people who tell me GMT (because they think that term means "the time in London") when they meant BST (because in summer, London doesn't operate on GMT).
Re: AWS us-east-1 down
#237For those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.
The fact that which timezone is used in the announcement is a sign of progress... AWS announced it pretty quickly, gave nice updates, and seems to have fixed the problem quickly enough. I'm interested to see the postmortem...
Re: AWS us-east-1 down
#238Earlier quoted context omitted.
3 years ago, when I started work for my current employer, I noticed in Slack that everyone was reckoning time in "Standard Time" year-round. Now imagine my chagrin because I live in Arizona, and "Mountain Standard Time" does not change for DST. Therefore, all my coworkers were citing nonsensical, nonexistent time zones and it was messing up my ability to convert back and forth. Come to find out that this was some sor…
How does this generate off-by-one errors? I am also part of a company with employees in pretty much every timezone, but when they create a meeting the meeting invitation is programmed with the correct timezone so in my Calendar it always shows what time the meeting is going to be for me. I never even have to think what timezone the organizer is...
The off-by-one error occurs when you announce an event in Standard time but really mean Daylight time, or vice versa. While those local to the time zone will often automatically correct this mistake either consciously on unconsciously, those in other time zones (especially where Daylight time isn't used or is on a different schedule) will tend to rely on time conversion tools which will take a literal interpretation of the scheduled time and result in the person being an hour early or an hour late.
Re: AWS us-east-1 down
#239Mysterious lack of "AWS is bad for the internet because it is so centralized" dialog up in here. edit: for those that would downvote: HN _just_ yesterday: https://news.ycombinator.com/item?id=36295352 https://news.ycombinator.com/item?id=36295305
its just tired at this point. everyone knows, nobody seems to care. Another comment of mine in this thread asks the question if you can excuse downtime of your service due to AWS outages. Consensus seems to be: yes which is a pretty huge deal, well worth the insane cost increase of AWS by itself. No other hosting provider would grant you such an excuse. I would weep for the centralised future of the internet, but its…
Even if people _do_ care, there isn't much to do about it.
Re: AWS us-east-1 down
#240Earlier quoted context omitted.
What I did once I was in the position of _having_ to provide that level of support, was to run the pipeline in a third region, different from the "prod" ones. That way, worst case you can't do deployments during the outage... Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run
> It was discarded bc you never know where that would actually run Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.
Yeah, that's why we went with a third region instead. But, at the end of the day, if _only_ changes are affected for a couple of hours, that wouldn't impact the service that much