771–780 of 870 posts
Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.
Earlier quoted context omitted.
Seems like you want insurance. As with the hospital bill you'd generally be paying a bunch of extra money for your health insurance plan to not get stuck with the bill. Not sure that exists for businesses, but I'd expect you'd need to go shopping separately if you want that. Seems like a good business idea if it doesn't exist.
Independent 'a service was down' insurance isn't the same though. It is important for the cost to come out of the provider's pocket, thus giving them a huge financial incentive to not be down. Having that incentive in place is the most important part of an SLA.
Earlier quoted context omitted.
When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".
Welp, as a new grad there, I had brought down one very important database server on a Sunday night (a series of really unfortunate events). Multiple senior DBAs had to be involved to resuscitate it. It started functioning normally just a few hours before market open in HK. If it was any later, it would have been some serious monetary loss. Needless to say, I was sweating bullets. Couldn't eat anything the entire day…
So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…
I think of it like thinking of trying to avoid the risks of flying. Sure, there have been many crashes in the last decade itself, let alone the last century. But we have only learned from them, and the future is thus much safer thanks to those experiences.
So I deal with it basically by resigning myself to the idea that I must keep backup of vitally important information, and the rest will no matter, in such an event.
Earlier quoted context omitted.
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
Several years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a small subset of queries for about an hour. As part of the postmortem I had to calculate how many people likely tried to interact with it while it was broken. It was a lot and really made me realize the magnitude of an otherwise seemingly small production failure.…
Earlier quoted context omitted.
I've never seen an SLA that compensates for anything more than credit off your bill. I can't imagine a service that pays for loss of productivity, one outage and the whole company could be bankrupt. If your business depends on a cloud service for productivity you should have a backup plan if that service goes down.
I haven't seen one (at least for a SaaS company) that will compensate for loss of productivity/revenue etc, but something like Slack's SLA[0] seems like it's moving in the right direction. They guarantee a 99.99% uptime (max downtime of 4 min/22 seconds per month) and give 10x credits for any downtime. Granted, there's probably not many businesses that are losing major revenue because slack's down for half an hour, b…
They won't show up on automated systems aimed at SMEs, but anybody taking out an "enterprise plan" with tailored pricing from a SaaS, will likely ask for tailored SLA conditions too (or rather should ask for them).
Earlier quoted context omitted.
For a large or v large tech co you should probably be aggregating logs to a centralised location that doesn't require access to production systems in this way. Stack dumps should also be collected safely off-system if necessary. Perhaps my industry is a little more security conscious (I don't know which industry you're talking about), but this doesn't seem like good practice.
Let me be clear, I agree it should not be normal to SSH into a prod box. Our logs are centrally aggregated. But it’s one thing to say it’s not normal, but quite another to say engineers shouldn't have access , because I totally disagree with that.
Earlier quoted context omitted.
Irrelevant to the discussion, but I just wanted to say thank you for the categorized list of users I can follow on your profile!
Which tool do you use to follow users on HN?
Another workflow, though cumbersome, is: Search for a username on hn.algolia, select "comments" and "past months" as filters, then press enter.
Ex: https://hn.algolia.com/?dateRange=pastMonth&query=nostrademo...
Earlier quoted context omitted.
Would they have less impact? Or would it have the same impact, just distributed across many more outages? You can rely on Google outages being very few and far between, and recovering pretty fast. For the benefits you get from such a connected ecosystem, I'm not sure anyone is net positive from using a variety of different tools rather than Google supplying many of them.
Compare closing down one road for repair a day per year to closing down all roads one day a year.
Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.
In my defence, the cert was not labeled properly, nor was it used properly, and there was no documentation. It took us 2 days to create a new cert and apply it to our software and deliver it to the customer. Those were 2 days I'll never get back. However, when I was finished the process was documented and the cert was labeled, so I guess its a win.