91–100 of 152 posts
https://status.heroku.com/incidents/1892 - it appears Heroku is being particularly affected. We've had multiple sites on multiple accounts go down in the past few minutes. EDIT T16:31Z: It appears Heroku has failed over their dashboard, but dynos are still failing to come online. We had assumed that they had multi-region failovers for their customers. Incredibly disappointing.
That must be why reddit and twitter are failing on me.
Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…
I'm interested in any evidence to back up my impression if anyone has bothered to do the proper data gathering.
(Aside, stink eye on whoever made a breaking change over a holiday weekend, if this turns out not to be random.)
My client's Heroku instances are online, thankfully.
Can anyone here speak to their experience with the Ohio region? I'm considering leaning on that more and more.
Has anyone else noticed that there seems to never be outages in us-east-2 and somehow everyone keeps putting instances in -1? Why?
Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…
Maybe just because it's been around the longest, but my impression is that us-east-1 seems to have more than its fair share of outages. Personally, for my single-region applications focusing on US customers, I go with us-east-2. Knock on wood. I'm interested in any evidence to back up my impression if anyone has bothered to do the proper data gathering. (Aside, stink eye on whoever made a breaking change over a holid…
Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…
For us this is exactly the correct approach. We could have spent millions of dollars and thousands of man hours hardening things to be resilient to single region outages. But for what? We aren't GE or Google. If our conference line goes down for 2-3 hours per year because we don't have apocalypse-proof infrastructure, literally nothing bad happens to our business. In this exact outage we are discussing, all of my cow…
Specific providers aside, there's more complexity involved in a large cloud provider's infrastructure and much more that can go wrong as a result. Having a code update, or some orchestration issue from your infrastructure provider be potential points of major outages are huge and unnecessary risks. You don't need that much scale, just utilizing enough resources to fill up a few whole physical machines for a few hundred dollars a month. Add some globally distributed BGP Anycast DNS and database replication and you have enough redundancy to withstand most of the worst major infrastructure failures.
I would understand if AWS was super simple and convenient, but these days the learning curve seems far greater than setting up the above described bare metal solution. While being almost an order of magnitude more expensive for the equivalent amount of resources.
How did we end up here? Does brand recognition just trump all technical and economic factors, or what am I missing?
Disclaimer: I run a bare metal hosting provider
Earlier quoted context omitted.
We cannot even restart/turn off the dynos or get into the dashboard to turn off and kill our background tasks for some of our clients.
Is it best practice to run the dashboard and the cloud service in the same region of the same cloud provider?
As a PaaS I would think that they would run a high availability cluster on at least 2 multiple regions so that they would have a mechanism in place for events like these. I know it's expensive, but if you charge 250 for 2.5GB of RAM I believe you would have enough money to cover it. I also think as you hinted that they should separate services across different regions..
Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…
This should always be a business calculation but I do think you should note that there’s at least an order of magnitude difficulty increase between multiple AZs and regions, especially if you’re using services like RDS where it’s designed in, so I’d consider that a solid bridge step. I’m trying to put some numbers into that, I’ve been running a relatively well trafficked website in multiple AZs since 2011. We had ~20…