The two Google Cloud networking incidents are: Incident #19008 began at 2019-06-02 12:48. https://status.cloud.google.com/incident/cloud-networking/19... Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19... Times are US/Pacific
Looks like they are having trouble updating their statuses. 19008 was supposed to be updated over an hour ago. Meanwhile, 19009 has the same comment posted three times. I'm guessing internal tools are barely working at best.
I haven’t written a status page in a while, but the rest of my infrastructure starts freaking out if it hasn’t heard from a service in a while. Why doesn’t their status page have at least a warning about things not looking good?
In my experience public status pages are "political" and no matter how they start tend to trend towards higher management control in some way... that leads to people who don't know, aren't in the thick of it, don't understand it, and / or are cautious to the point that it stops being useful.
Not only political, but with SLAs on the line they have significant financial and legal consequences as well. Most managers are probably happier keeping the ‘incident declaring power’ in as few a hands as possible to make sure those penalty clauses aren’t ever un-necessarily triggered.
Nest is down for me right now. It's amazing how far-reaching outages can be these days.
Code reuse is a wonderful thing, until it's not.
Code reuse? Couple if years ago some started to call servers "cloud" but the cloud is just that: computers managed by someone else. If the whole service layer relies on Three big companies, there is a problem..
What I've realized from this: Google doesn't have an official status page for GCP. There are a few unofficial ones, but nothing official that I could find.
Solutions based on third-party butts have essentially two modes: the usual, where everything is smooth, and the bad one, where nothing works and you're shit out of luck - you can't get to your data anymore, because it's in my butt, accessible only through that butt, and arguably not even your data. With on-prem solutions, you can at least access the physical servers and get your data out to carry on with your day whi…
Assuming you have data that is tiny enough to fit anywhere other than the cluster you were using. Assuming you can afford to have a second instance with enough compute just sitting around. Assuming it's not the HDDs, RAID controller, SAN, etc which is causing the outage. Assuming it's not a fire/flood/earthquake in your datacenter causing the outage. ...etc.
Ah, yes, I will never forget running a site in New Orleans, and the disaster preparedness plan included "When a named storm enters or appears in the Gulf of Mexico, transfer all services to offsite hosting outside the Gulf Coast". We weren't allowed to use Heroku in steady state, but we could in an emergency. But then we figured out they were in St. Louis, so we had to have a separate plan for flooding in the Mississippi River Valley.
You obviously don't recall the early years of AWS. Half of internet would go down for hours.
Back when S3 failures would take town Reddit, parts of Twitter .. Netflix survived because they had additional availability zones. I can remember the bigger names started moving more stuff to their own data centers. AWS tries to lock people in to specific services now which makes it really difficult to migrate. It also takes a while before you get to the tipping point where hosting your own is more financially viable…
I am not sure if a single S3 outage pushed any big names into their own "datacenter". S3 has still the world record of reliability that you cannot challenge with your inhouse solutions. You can prove it otherwise. I would love to hear a solution that has the same durability, avabiality and scalability as S3.
For the downvoters, please just link here the proof if you disagree.
I see a lot of Nest complaints, isn't there any security issue if Nest goes down?
I see people complain about Nest and the only thing I can think of is "what on earth where you thinking to have a door or thermostat that doesn't function without internet?!"
I live in a house with a Nest thermostat and Nest Secure and Nest x Yale door locks. The AC is on just fine, and the door unlocked just fine. (The door lock doesn't require an internet connection, unless you've enabled privacy mode for some reason.)
I would question why they are doing a rollout on a weekend?
A config push on a weekend seems pretty unlikely. Given that it's apparently a network congestion issue and showed up on a weekend, my guess is that it's probably a bizarre networking hardware failure like the one that took out CenturyLink[1] last December.
I haven’t written a status page in a while, but the rest of my infrastructure starts freaking out if it hasn’t heard from a service in a while. Why doesn’t their status page have at least a warning about things not looking good?
In my experience public status pages are "political" and no matter how they start tend to trend towards higher management control in some way... that leads to people who don't know, aren't in the thick of it, don't understand it, and / or are cautious to the point that it stops being useful.
Same with most corporate twitter feeds. I’d like to follow my public transit/airport/highway authority, but it’ll be 10 posts about Kelly’s great work in accounting for every service disruption.
And No, I don’t want to install a separate app to get push notifications about service disruptions for every service I use.