81–88 of 88 posts
That would explain why my ETLs are failing at 3am in the morning !!!
Earlier quoted context omitted.
US schools don't all start on the same day though- its pretty staggered with some starting in early-mid august, and most in the Northeast start right after Labor day.
It's still probably a normalish distribution
And if this happened last year too, you would think this would be on top of the list of things to watch for next year and add capacity for. Amazon and Walmart start planning and drilling now for their holiday season.
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
That's an interesting theory because the timing does correlate. A lot of people would immediately dismiss it because Google has the resources to scale up. But having resources doesn't guarantee someone actually turns the knob that increases the number of instances. (Whether automatic or manual, the adjustment could be too slow to match an unanticipated spike in demand.) But there's another reason I don't think that's…
IMO it's not really about having the resources to scale, but the unpredictable emergent behaviours which can happen when the load profile suddenly changes
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997
Every year around the same time people have to work on Perf (internal performance review), maybe people were more focused on that rather than keeping the systems up.... or maybe they needed to push the latest update to be included in their perf...
That would explain why my ETLs are failing at 3am in the morning !!!
If it makes you feel any better; same thing happened to me. Ugly Frankenstein process to get our data out of a third party vendor who can only e-mail us reports failed half-way through its run, and I was about to spend hours digging through to see why until I saw this.
I have 5 tabs open with emails written ready to send at some point. Had issues saving drafts, too.
Earlier quoted context omitted.
Every year around the same time people have to work on Perf (internal performance review), maybe people were more focused on that rather than keeping the systems up.... or maybe they needed to push the latest update to be included in their perf...
I like this theory too - but is performance review this week?
Earlier quoted context omitted.
Not sure if this is still the case, but if you did this a couple of times, your account data would be permanently migrated to an instance with more CPU and RAM allocated - you'd also be in with all the other badly behaved accounts, so reliability goes down lots. The benefit was much quicker complex searches, and being able to bulk label or delete emails without it taking minutes or hours. Don't believe me how slow it…
I find it hard to believe that Gmail will always serve certain users from the same machines, especially in this day and age, with “cattle, not pets” and ephemeral containers. I’m sure they have machines that are only used to serve G Suite and Google One customers, and maybe some other VIPs, but regular heavy users? It sounds like an urban legend to me.