91–100 of 186 posts
I do laugh at people that say "Using the cloud means less downtime than your own server", then things like this come along :D
It’s still less downtime than your own server and you have hundreds of engineers 24/7 ready to diagnose and fix issues
Of course I'm not trying to run a massively scalable service coping with millions of customers, because I don't need that.
Earlier quoted context omitted.
Googler here, not speaking on the behalf of the company, my opinions are my own People do absolutely NOT get fired over incidents. Making mistakes is human. An incident will prompt a review of the systems and safeguards in place to prevent such an incident, much like an airline incident investigation - basically "somebody fat-fingered it" is never the answer, postmortems are always blameless EDIT: now that I think of…
> Googler here, not speaking on the behalf of the company, my opinions are my own Why do employees at big tech names (FAANG et al.) are so often so cautious as to include this as a foreword everywhere? Twitter bios are full of that, for instance. It is crazy to me; who would expect anything else that our opinions being your own and nothing more? Who would expect that your word (with all due respect) is worth anything…
Earlier quoted context omitted.
Well, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made. Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.
> logical conclusion regarding the spiky yearly activity Why is it logical? There’s tons of changes being deployed at all times, at all large companies. Across products, verticals, everything - hundreds of changes at any given point. Some of these changes can introduce hard-to-predict bugs in globally distributed systems. Most of the time, external users don’t even notice before they’re fixed. Like another commenter…
Of course, I wouldn't go as far as to call this a "logical conclusion" as the evidence I've seen is very slim.
I do laugh at people that say "Using the cloud means less downtime than your own server", then things like this come along :D
It’s still less downtime than your own server and you have hundreds of engineers 24/7 ready to diagnose and fix issues
When you have a "simple" (already quite complex) BGP(TCP(HTTP)) tunnel, chances are things just work and it's easy enough to diagnose issues. Between anycast, auto-scaling, and WAFs (among others) you have added so many layers of complexity that the chance of a "random" error somewhere in the stack has dramatically increased and diagnosis is close to impossible.
It used to be simple that web services are either up or down. Now it's not so obvious anymore, and that's definitely not being reflected in the 5 9's SLAs falsely promised by cloud providers. I can say with confidence most selfhosted systems i've been close to have much better uptime than modern cloud services and are much cheaper to service and maintain.
Maybe they are just upset because they got fined. It's common by big companies.
Earlier quoted context omitted.
The other comments already explained it, but I'm wondering how you haven't come across this 'saying' before. It's so overused and also cheesy in my opinion.
Lol maybe if people actually did spend millions training their people up front we could do better?
$140 billion dollars. On training.
On the one hand... you know what, I'd love to work in an environment like that. Seriously.
On the other hand... what's the argument you make to the CFO in support of this? Honest question, interested to hear answers.
Seems that google is not anymore the company it used to be advertised. The myth prevails though. It’s a pity as some of its services are frankly among the best, and I depend on them.
Do you mean about the "Don't be evil" slogan it started with? Even back then, it was a pretty obvious move for any movie villain.
Or do you mean you actually bought into the reliability promises of cloud providers thinking downtime would not exist anymore?
Earlier quoted context omitted.
It’s still less downtime than your own server and you have hundreds of engineers 24/7 ready to diagnose and fix issues
Really, downtime of google so far this year: More than 0 minutes. Downtime of my own server: 0 minutes. Of course I'm not trying to run a massively scalable service coping with millions of customers, because I don't need that.
Earlier quoted context omitted.
Perf has been over for almost a month now, and the evaluation period was over more than two months ago.
> and the evaluation period was over more than two months ago Excellent time to slack off a bit, make a few mistakes, then come next eval you can point to a marked improvement over the intervening six-to-nine months!
Earlier quoted context omitted.
It’s still less downtime than your own server and you have hundreds of engineers 24/7 ready to diagnose and fix issues
This is so misinformed by industry propaganda. Modern cloud services are very often unavailable from a region without any apparent reasons. Or the service appears to be available but some specific feature doesn't work. Or it's available but just really slow or dropping packets. When you have a "simple" (already quite complex) BGP(TCP(HTTP)) tunnel, chances are things just work and it's easy enough to diagnose issues.…
Firstly, I think that the GP comment ever so slightly misapplies the localized awareness available in smaller environments to the cloud. Yes, there are tons more engineers to look at stuff, but those engineers are a) already bogged down keeping up with internal infra, politics and bureaucratic machinery, and b) quite some distance further away from actual errors occurring on the ground because everything's aggregated to the hilt so the stats remain comprehensible at the higher scale.
I also agree that the newer stuff that is less mature has an order of magnitude more intrinsic leaky abstractions than old designs, which I do think were more hygienic and espoused more effective separation of concerns than what is used today.
Also, not to nitpick, but the "classical" canonical definition would probably look closer to BGP(TCP(TLSv1.3(HTTP/1.1))), with the modern equivalent being BGP(UDP(HTTP2)) and the future being BGP(UDP(QUIC)). I do agree that the rapid consolidation of HTTP and TLS, without wide-scale awareness and slow, methodological development of general introspection tooling, does make things net worse in general. I suspect infra will still be using HTTP/1.1 for a long time into the future until this materially changes.