There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
Millions of people are searching simultaneously at google.com or youtube.com but servers are not crashing. Issue is not traffic overload but something else.
Not sure if this is still the case, but if you did this a couple of times, your account data would be permanently migrated to an instance with more CPU and RAM allocated - you'd also be in with all the other badly behaved accounts, so reliability goes down lots. The benefit was much quicker complex searches, and being able to bulk label or delete emails without it taking minutes or hours. Don't believe me how slow it…
I find it hard to believe that Gmail will always serve certain users from the same machines, especially in this day and age, with “cattle, not pets” and ephemeral containers. I’m sure they have machines that are only used to serve G Suite and Google One customers, and maybe some other VIPs, but regular heavy users? It sounds like an urban legend to me.
Gmail accounts will sometimes be automatically "hospitalized" -- assigned more than the usual amount of resources because for some reason they are chronically behind or growing without bound -- or "jailed" -- moved into isolation along with other bozo accounts, to keep from disturbing normal people's accounts. Not a legend.
Millions of people are searching simultaneously at google.com or youtube.com but servers are not crashing. Issue is not traffic overload but something else.
Primarily cache hits with no state.
This. And to expand on it:
for Gmail, you have to log in. It has to know it is you. It has to know that everything it is serving you is only for you. It has to retain copies of documents that -- as far as it knows -- are unique to you or whomever you share with. It has to do all this while keeping that information safe from other people who might want to take a look at it.
And that's just the items I can think of in real time while typing.
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
Every year around the same time people have to work on Perf (internal performance review), maybe people were more focused on that rather than keeping the systems up.... or maybe they needed to push the latest update to be included in their perf...
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
US schools don't all start on the same day though- its pretty staggered with some starting in early-mid august, and most in the Northeast start right after Labor day.
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
US schools don't all start on the same day though- its pretty staggered with some starting in early-mid august, and most in the Northeast start right after Labor day.
Incident says it's affecting anything that uses Cloud Storage, which I'm guessing is most things. https://status.cloud.google.com/incident/zall/20007 Edit: New incident number https://status.cloud.google.com/incident/zall/20008
Well this happens if you rely only on 3rd party providers. My email server is still running without any issues and without any indecent for last 3 years (system reinstall). Surely no one can send me emails now that gmail is down as most of world is relying on single point of failure, but this is another story. Moral of this story is - always own your mission critical infrastructure.
> Moral of this story is - always own your mission critical infrastructure.
Sure, if you have the time and money to own it properly. How far do you need to go to say you own it? Multi-regional servers located on properties that you own?
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
That's an interesting theory because the timing does correlate.
A lot of people would immediately dismiss it because Google has the resources to scale up. But having resources doesn't guarantee someone actually turns the knob that increases the number of instances. (Whether automatic or manual, the adjustment could be too slow to match an unanticipated spike in demand.)
But there's another reason I don't think that's the explanation. Gmail has 1.5 billion active users[1]. Millions of students logging in at the same time sounds like a lot, but if Gmail has 100 million more active users today than yesterday, that's not even a 10% increase!
Not sure if this is still the case, but if you did this a couple of times, your account data would be permanently migrated to an instance with more CPU and RAM allocated - you'd also be in with all the other badly behaved accounts, so reliability goes down lots. The benefit was much quicker complex searches, and being able to bulk label or delete emails without it taking minutes or hours. Don't believe me how slow it…
I find it hard to believe that Gmail will always serve certain users from the same machines, especially in this day and age, with “cattle, not pets” and ephemeral containers. I’m sure they have machines that are only used to serve G Suite and Google One customers, and maybe some other VIPs, but regular heavy users? It sounds like an urban legend to me.
Not the same machines, but different groups or machines (pools, shards, farms, whatever). They'll certainly be able to move you around to balance the pools, or to decomission pools, or to put you in a pool with primary data closer to where you usually access from etc. Grouping by behavior makes sense too --- separating heavy and light users makes a lot of sense, you can serve a lot more light users from one pool, and the heavy users won't impact their service.
Good luck with that - read GSuite's terms, they, Google, define what an outage is, not the customer.
The linked terms say: > "Downtime" means, for a domain, if there is more than a five percent user error rate. Downtime is measured based on server side error rate.
Which is fantastic if the error causes the server side not to be able to log errors