Live data from Hacker News

Google outage – resolved

news.ycombinator.com

791–800 of 870 posts

Re: Google outage – resolved

#791
post #292

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

Nope, especially considering the implications of this, with the amount of people working remotely. Google Meet, Classroom, etc. are down. This is probably literally costing billions every minute just in loss of productivity.

I never understood this type of calculation as it implies that time is directly converted into money. However, I struggle to come up with an example for this. Even the most trivial labor cases like producing paperclips don't seem to be directly converting time into profit: even you will make 10k units instead of 100k this hour, you don't sell them immediately. They bring revenue to the firm via a long chain of convoluted contracts (both legal and "transactional") which are very loosely coupled to the immediate output.

Nothing is operating at minute margins unless it's explicitly priced on a minutely basis, like a cloud service. Even if a worked on a conveyor belt can't produce paperclips without looking at Google Docs sheet all the time, this will be absorbed by the buffers down the line. And only if the worker will fail to meet her monthly target due to this, loss of revenue might occur. But in this case the service has to be down for weeks.

In case of more complex conversions of time into money, like in the most of intellectual work, this is even less obvious that short downtimes will cause any measurable harm.

Re: Google outage – resolved

#792
post #761

Earlier quoted context omitted.

Which tool do you use to follow users on HN?

There used to be hystry.com [0] but it isn't functional anymore. Another workflow, though cumbersome, is: Search for a username on hn.algolia , select "comments" and "past months" as filters, then press enter. Ex: https://hn.algolia.com/?dateRange=pastMonth&query=nostrademo... [0] https://news.ycombinator.com/item?id=71827

Thanks for sharing those tips!

Re: Google outage – resolved

#793
Anyone experiencing lost emails? I'm not sure if it's actually me messing things up or because of the Gmail outage, I couldn't find an extremely important email and I've tried archive, trash, deleted, spam, sent boxes..

Re: Google outage – resolved

#794
post #424

Earlier quoted context omitted.

What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…

I've been happy with using Hosted Exchange on Microsoft. I own the domain so ultimately I can point the DNS to some other provider. Outlook stores the mails locally so I have a backup. I think the most important thing about E-Mail is to receive future E-Mails and not look at historical ones. In the end you can always ask the receipent to send you a copy of the email conversation - if you dont own the domain it get mu…

For That's _much_ better than trying to host my own email server.

[0] https://docs.microsoft.com/en-us/office365/servicedescriptio...

Re: Google outage – resolved

#795
post #744

Earlier quoted context omitted.

Sadly a lot of managers don't see it this way, they'd rather assign blame.

Depending on the company's culture, that calculus can vary. If management start firing subordinates for making mistakes, then what should be done to management if they fail to account for human error, resulting in multimillion-dollar losses?

Well, why, good Sir - don't they usually get bonuses?

Re: Google outage – resolved

#796

So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?

> Of course it will, - at least, it better - but what if it doesn't?

Same question for non-cloud.

Re: Google outage – resolved

#797
post #424

Earlier quoted context omitted.

What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…

I'm running my own mail server, and I think anyone who has some experience with Linux should be able to do the same in a day or two. Once it's set up it just works. You can still use Gmail and fall back to connecting directly to your server if Gmail is down. Some mails might be flagged as spam if the IP/domain has no reputation, but that quickly passes, at least that's my experience.

I guess something highly resilient would be - say - a mailserver on a rented VM replicating to two cloud providers via a service mesh.

Nice and simple! :D

Re: Google outage – resolved

#798
post #622

When I tried to login it said: 'this account does not exist'. So my first thought was some algorithm made a mistake and my account got deleted for no reason. I already imagined the only solution now was to write a medium post and hope it gets some traction on hackernews and google support steps in. Thinking to myself I was an idiot for knowing all this and still thinking it wouldn't happen to me. And even though it t…

Those hosting with their own domains on Google mail had an identical experience, if it makes you feel differently.

Yep, but losing access to my emails isn't that big of a deal, losing access to my email address is much worse. Especially since it's also my login on a lot of other services.

Re: Google outage – resolved

#800
post #775
post #766

Earlier quoted context omitted.

Several years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a small subset of queries for about an hour. As part of the postmortem I had to calculate how many people likely tried to interact with it while it was broken. It was a lot and really made me realize the magnitude of an otherwise seemingly small production failure.…

Presumably there were more failures than a single engineer could've been responsible for here.

Its absolutely possible, the worst AWS outage was caused by one engineer running the wrong command [0].

"This past Tuesday morning Pacific Time an Amazon Web Services engineer was debugging an issue with the billing system for the company’s popular cloud storage service S3 and accidentally mistyped a command. What followed was a several hours’ long cloud outage that wreaked havoc across the internet and resulted in hundreds of millions of dollars in losses for AWS customers and others who rely on third-party services hosted by AWS."

[0] - https://www.datacenterknowledge.com/archives/2017/03/02/aws-...

Post reply on HN