looking forward to reading the postmortem
Google outage – resolved
671–680 of 870 posts
Re: Google outage – resolved
#672So to easily fix such down time, i have alerts setup for website, server and API using https://simpleops.io/
After that I realised how frequently you can get down time
Re: Google outage – resolved
#6734:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…
Oh man, you're right. Bloated gmail loaded instantly. What's going on? It's loading almost 2x to 3x faster.
I always thought gmail being slow is because of me using firefox but now it's surprisingly snappy. What the hell is going on?
Re: Google outage – resolved
#674If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...
SLAs are largely bullshit.
Re: Google outage – resolved
#675If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...
If you click on the red status dots, it has report with timing.
Re: Google outage – resolved
#676If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...
SLAs are largely bullshit.
why?
Re: Google outage – resolved
#677Earlier quoted context omitted.
That's trillion not billion
https://en.wikipedia.org/wiki/Billion A billion is a number with two distinct definitions: - 1,000,000,000, i.e. one thousand million, or 10^9, as defined on the short scale. This is now the meaning in both British and American English. - 1,000,000,000,000, i.e. one million million, or 10^12, as defined on the long scale. This is one thousand times larger than the short scale billion, and equivalent to the short scal…
This is a financial discussion though so:
https://www.worldometers.info/gdp/gdp-by-country/
World's GDP is $80,934,771,028,340 (nominal, 2017).
https://www.wolframalpha.com/input/?i=%2480%2C934%2C771%2C02...
$80.93477102834 trillion
Nobody would argue world GDP is anything billion, that's crazy.
Re: Google outage – resolved
#6784:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…
If this phenomenon is actually real instead of just perception then I'd guess it is down to reduced demand of some short. Some possibilities: a) users haven't all come back yet b) Google is throttling how fast users can access services again to prevent further outages c) to reduce load, apps have features turned off (which might make things directly faster on the user's end or just reduce load on the server side)
At Google's scale, I'd expect it to be all of the above.
I hope they make their learnings, post-mortem, etc. public so that we can all learn from it.
My engineer hat is saying - "damn, I wish I was part of fixing this outage at their scale."
My product owner hat is saying - "Aaaaaaaaaaaaaaa......Aaaaaaaaaaaaaaa...."
:D
Re: Google outage – resolved
#679Earlier quoted context omitted.
If you alone were able to do it, then the system was designed badly. The bigger the impact, the more robust it has to be to prevent accidents.
The big mistake in the system is that everyone in the world is relying on Google services... These problems would have less impact with a more diverse ecosystem.
Would they have less impact? Or would it have the same impact, just distributed across many more outages?
You can rely on Google outages being very few and far between, and recovering pretty fast. For the benefits you get from such a connected ecosystem, I'm not sure anyone is net positive from using a variety of different tools rather than Google supplying many of them.
Re: Google outage – resolved
#680Earlier quoted context omitted.
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
I remember how one of our engineers had his docker daemon connected to production instead of his local one and casually did a docker rm -f $(docker ps -aq) . Same thing happened to me but with CI, which felt bad enough already.
No Engineer should have production access from their workstation. Period.
source: am Engineer =).