Live data from Hacker News

Gmail and Google Drive Outage

google.com

51–60 of 88 posts

Re: Gmail and Google Drive Outage

#51
post #22

I'm guessing this outage will allow GSuite customers to claim Service Credits under the SLA - does anyone have any experience with doing so? Google's documentation is lacking in detail[0]. [0] https://gsuite.google.com/intl/en/terms/sla.html

Follow up to that: Does anyone know if there are tools to help customers collect and submit data to their their vendors for SLA credits?

I’d like to add something like that to StatusGator but I’m unsure if there’s a market.

Re: Gmail and Google Drive Outage

#52

There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997

Google does a once per year disaster recovery training... They do things like deliberately turn off datacenters with no warning. Sometimes failover systems don't work as intended. Was that this week?

"dirt" - https://www.usenix.org/conference/lisa15/conference-program/...

Re: Gmail and Google Drive Outage

#53

There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997

Millions of people are searching simultaneously at google.com or youtube.com but servers are not crashing. Issue is not traffic overload but something else.

Drive and Gmail are not the same thing as search. The bottlenecks are different, the architecture and problem spaces aren't the same either.

Re: Gmail and Google Drive Outage

#54

Earlier quoted context omitted.

But I'm sure similar infrastructure architecture was applied to gmail.com as it was to google.com and youtube.com. And similar concepts of maintaining by sysadmins are practiced.

Hah... Press and hold the F5 key on your keyboard for 2 minutes while on gmail.com. You will get a "service unavailable" error. About 500 other people whose data happens to be cohosted with you will also get the same error, and all of you will be unable to send or receive email, even by IMAP, for about 10 mins while your particular corner of the data store is restarted and the data integrity checked. That doesn't hap…

Ok, I definitely want to know how you discovered that... (and found one of those 500 people to verify?)

Re: Gmail and Google Drive Outage

#55
post #22

I'm guessing this outage will allow GSuite customers to claim Service Credits under the SLA - does anyone have any experience with doing so? Google's documentation is lacking in detail[0]. [0] https://gsuite.google.com/intl/en/terms/sla.html

Good luck with that - read GSuite's terms, they, Google, define what an outage is, not the customer.

Re: Gmail and Google Drive Outage

#56
post #24

The status from Google Cloud status page offers a bit more technical details of what happened: "We are experiencing an issue with Google Kubernetes Engine (GKE) clusters using node auto-provisioning becoming stuck during node version upgrades. Node auto-upgrades have been disabled temporarily." https://status.cloud.google.com/

Does it mean, that Gmail is hosted on GKE?

Re: Gmail and Google Drive Outage

#57

Earlier quoted context omitted.

Hah... Press and hold the F5 key on your keyboard for 2 minutes while on gmail.com. You will get a "service unavailable" error. About 500 other people whose data happens to be cohosted with you will also get the same error, and all of you will be unable to send or receive email, even by IMAP, for about 10 mins while your particular corner of the data store is restarted and the data integrity checked. That doesn't hap…

Ok, I definitely want to know how you discovered that... (and found one of those 500 people to verify?)

Not sure if this is still the case, but if you did this a couple of times, your account data would be permanently migrated to an instance with more CPU and RAM allocated - you'd also be in with all the other badly behaved accounts, so reliability goes down lots. The benefit was much quicker complex searches, and being able to bulk label or delete emails without it taking minutes or hours.

Don't believe me how slow it is on a regular instance? Try going to "All mail", selecting all of your emails, and applying a label to them all. In my experience, it can only label about 50 mails per second, so it can take hours to do them all. It will keep going if you quit the browser, but will stop if the gmail devs do a software update, which they seem to do on usually tuesdays, but never fridays or the weekends.

Re: Gmail and Google Drive Outage

#58

Earlier quoted context omitted.

Ok, I definitely want to know how you discovered that... (and found one of those 500 people to verify?)

Not sure if this is still the case, but if you did this a couple of times, your account data would be permanently migrated to an instance with more CPU and RAM allocated - you'd also be in with all the other badly behaved accounts, so reliability goes down lots. The benefit was much quicker complex searches, and being able to bulk label or delete emails without it taking minutes or hours. Don't believe me how slow it…

I get brief gmail outages once every several months. Bulk labelling is really fast. I wonder if this is why...

Re: Gmail and Google Drive Outage

#59
post #55
post #22

I'm guessing this outage will allow GSuite customers to claim Service Credits under the SLA - does anyone have any experience with doing so? Google's documentation is lacking in detail[0]. [0] https://gsuite.google.com/intl/en/terms/sla.html

Good luck with that - read GSuite's terms, they, Google, define what an outage is, not the customer.

The linked terms say:

> "Downtime" means, for a domain, if there is more than a five percent user error rate. Downtime is measured based on server side error rate.

Re: Gmail and Google Drive Outage

#60

Earlier quoted context omitted.

But I'm sure similar infrastructure architecture was applied to gmail.com as it was to google.com and youtube.com. And similar concepts of maintaining by sysadmins are practiced.

Hah... Press and hold the F5 key on your keyboard for 2 minutes while on gmail.com. You will get a "service unavailable" error. About 500 other people whose data happens to be cohosted with you will also get the same error, and all of you will be unable to send or receive email, even by IMAP, for about 10 mins while your particular corner of the data store is restarted and the data integrity checked. That doesn't hap…

Interesting. I held F5 down for less than a minute and I got an "Unusual usage - account temporarily locked down" message. Disappeared after a few seconds pause though...
Post reply on HN