How many folks actually use multi-region deployments with automatic failover (e.g. latency based routing in route 53)?
The difficulty of making this work in practice is pretty high. It also isn't cheap, so I would guess not many.
AWS us-east-1 down
241–250 of 331 posts
Re: AWS us-east-1 down
#242Mysterious lack of "AWS is bad for the internet because it is so centralized" dialog up in here. edit: for those that would downvote: HN _just_ yesterday: https://news.ycombinator.com/item?id=36295352 https://news.ycombinator.com/item?id=36295305
It's a mob mentality. Safety in numbers. "Oh well, my site is down but so is my neighbour's so nobody will be that mad about it."
Re: AWS us-east-1 down
#243Mysterious lack of "AWS is bad for the internet because it is so centralized" dialog up in here. edit: for those that would downvote: HN _just_ yesterday: https://news.ycombinator.com/item?id=36295352 https://news.ycombinator.com/item?id=36295305
Ok fine. Running your own datacenter in 2023 is incredibly risky. There's the upfront server cost and the ongoing maintenance cost. There's patches and staffing and disaster planning and all the other things that goes into it. Plus there's the cyberinsurance and protections and security components too.
Do you really think other (smaller) orgs can do a better job at hosting a datacenter than Amazon / Google / Microsoft / Cloudflare? They have some of the brightest minds in the industry working there, and they can price things at a much better price than anything you can build yourself.
Yes, I get it. All the computer processing power in a handful of actor's hands is probably not the most fantastic thing. However with the price of some cloud vendors compared to the DIY approach, it's hard for organizations to ignore.
If you really want to combat this, make the cost of running your own data center less. Reduce risk. Reduce the amount of money it costs for hiring good people or MSP's. Reduce the cost of acquiring and installing hardware.
Organizations pay attention to dollars so if you want the trend to shift, come up with a less costly alternative to the current cloud offerings.
Re: AWS us-east-1 down
#244Earlier quoted context omitted.
it's important to inform customers about the resiliency of their systems and let them pick how far they are going to invest for it. then you get to eat popcorn when stuff explodes. * single server event. $ * multi server event. $$ * single az event. $$$ * multi az event. $$$$ * global provider event. $$$$$ * cross provider event. $$$$$$ * alien invasion. $$$$$$$$$$$$$$
This should be logarithmic
The nice thing is that any graph without a unit can be log-scale - so in a way, it already is.
Re: AWS us-east-1 down
#245It doesn't matter if your infra is in another region, because there will almost always be transitive dependencies on us-east-1. IAM is deployed in us-east-1 and there will always be a transitive dependency on us-east-1
I have never had a production issue in other regions due to a us-east-1 outage. The worst that ever happened was I had to wait to update a Cloudfront distribution because the control plane (based in us-east-1) was down, but the existing configuration continued working fine throughout. I don't know what the architecture of IAM looks like, but somehow it's never suffered a global outage. AWS is really, really good at r…
I think the data plane is regional
Re: AWS us-east-1 down
#246Earlier quoted context omitted.
It's also usually extremely US-centric. Nobody outside of North America has any idea what "PDT" or "Mountain Time" means.
I mostly struggle with Irish Standard Time (used for DST in Ireland) and Indian Standard Time which have the same acronym. :( Thankfully, I learnt a long time ago to use ISO 8601 and UTC for dates and times. I still revert to PST/PDT if my audience is primarily left coast based.
> I mostly struggle with Irish Standard Time (used for DST in Ireland) and Indian Standard Time which have the same acronym. :(
Heh. After the first few instance of confusion, we switched to saying Bangalore time and Dublin time.
Re: AWS us-east-1 down
#247Earlier quoted context omitted.
3 years ago, when I started work for my current employer, I noticed in Slack that everyone was reckoning time in "Standard Time" year-round. Now imagine my chagrin because I live in Arizona, and "Mountain Standard Time" does not change for DST. Therefore, all my coworkers were citing nonsensical, nonexistent time zones and it was messing up my ability to convert back and forth. Come to find out that this was some sor…
This is why I always write ET instead of EDT/EST. I encourage everyone at my company to do the same. Easy way to eliminate errors while typing 1 less key stroke!
This is the way.
Re: AWS us-east-1 down
#248Earlier quoted context omitted.
Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names. Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.
Also report it using IATA time zones (America/Los_Angeles) at least in addition (I'd argue instead of) those abbreviations which are completely unstandardized and not unique.
If the world were fair, we’d be calling these Eggert time zones, as Paul Eggert (longtime tzdata maintainer until the copyright trolls came) invented them; but it isn’t.
(You probably still meant IANA the Internet org not IATA the aviation one.)
Re: AWS us-east-1 down
#249Both Vercel and Netlify went down with it.
I wonder what % of the internet went down because of the us-east-1 today.
Re: AWS us-east-1 down
#250Earlier quoted context omitted.
its just tired at this point. everyone knows, nobody seems to care. Another comment of mine in this thread asks the question if you can excuse downtime of your service due to AWS outages. Consensus seems to be: yes which is a pretty huge deal, well worth the insane cost increase of AWS by itself. No other hosting provider would grant you such an excuse. I would weep for the centralised future of the internet, but its…
Even if people _do_ care, there isn't much to do about it.
If people do care they could use other hosting providers such as Hetzner or OVH, no?