691–700 of 870 posts
Earlier quoted context omitted.
SLAs are largely bullshit.
why?
What if the majority of your users can access the service, but one of your BGP peers is not routing properly and some of your users are unable to access?
Earlier quoted context omitted.
I remember how one of our engineers had his docker daemon connected to production instead of his local one and casually did a docker rm -f $(docker ps -aq) . Same thing happened to me but with CI, which felt bad enough already.
No Engineer should have production access from their workstation. Period. source: am Engineer =).
Earlier quoted context omitted.
why?
Not OP, but how do you measure them? Let's say, for example, you can send and receive email, but attaching files does not work. Is the service up or down? What if the majority of your users can access the service, but one of your BGP peers is not routing properly and some of your users are unable to access?
So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…
You can still use Gmail and fall back to connecting directly to your server if Gmail is down.
Some mails might be flagged as spam if the IP/domain has no reputation, but that quickly passes, at least that's my experience.
4:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…
Everything is snappier for a while if you turn it off and then on again
Help! My Waymo taxi won't open the doors without me logging into the app. Its driving around in circles on route 500 and won't stop. /s - for now ;)
Earlier quoted context omitted.
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).
Earlier quoted context omitted.
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".
Earlier quoted context omitted.
That's what it was 15 years back. I was slightly concerned when Google Mail and Chat were integrated. Look where we are now. Right from "The battle of the red cliffs", we will never learn the disadvantage of lashing too many boats together.
But our tech right now is far more advanced than 15 years ago. We have IPFS, Blockchain, dat protocol today. I think it's possible to kill the giants. Even, Tim Berners-Lee want to decentralized the web: https://techcrunch.com/2018/10/09/tim-berners-lee-is-on-a-mi...
While it went OK-ish for the internet, we massively screwed it up for the web.
So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
They seemed to have figured out the hard parts already.