Live data from Hacker News

Google outage – resolved

news.ycombinator.com

861–870 of 870 posts

Re: Google outage – resolved

#861
post #749

Earlier quoted context omitted.

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

> At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases). It seems like a design flaw for actions like tha…

If it was indeed in-production, I'd never in a million years have had access rights to delete the stack. Those are gated just the way one would imagine they should be.

The service stack for the region (and not an entire region itself) looked like prod, but wasn't. It made me feel like shit anyway.

Re: Google outage – resolved

#862

Earlier quoted context omitted.

Don't worry they won't reply to this. Doesn't fit their narrative. Learning CS basics just isn't sexy enough for hacker news. Why learn anything over this.setState in React.

I love your response because I have a PhD in Computer Science. I do LeetCode as and CodeWars problems in the mornings for fun (I like coding and helps me avoid getting rusty) and I have been a hiring manager for over 10 years. But yeah, other than the "sexiness" of CS concepts it's only because of the "narrative" we want to push.

So you do think Leetcode is useful ? What's up with the flex ? Congratulations on the PhD and job, I guess ?. Looks like you could definitely invert a binary tree.

Re: Google outage – resolved

#863

So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?

Let's hope tailscale swoops in and creates a no-gimmicks, highly usable, private internet for everyone. They seemed to have figured out the hard parts already.

You mean like using Google GSuite for SSO? In the context of this authentication-related outage, it's a funny suggestion.

https://tailscale.com/kb/1013/sso-providers

Re: Google outage – resolved

#864

Earlier quoted context omitted.

I love your response because I have a PhD in Computer Science. I do LeetCode as and CodeWars problems in the mornings for fun (I like coding and helps me avoid getting rusty) and I have been a hiring manager for over 10 years. But yeah, other than the "sexiness" of CS concepts it's only because of the "narrative" we want to push.

So you do think Leetcode is useful ? What's up with the flex ? Congratulations on the PhD and job, I guess ?. Looks like you could definitely invert a binary tree.

Leetcode is good for some things, but in my experience is it not a good factor to filter Software Engineers (it is a good metric to filter candidates to form a team for the IOI or similar).

This is a response I wrote elsewhere:

I've been growing Engineering teams for the last 10 years as hiring managers in different startups. At some point in our startup we had those kind of HackerRank questions as filters.

The thing we realized is that those sort of interviews optimize to hire a specific type of very Jr Engineers who have recently graduated or are graduating from CS. That is because those are the people that have the time to churn these types of "puzzle" problems. Particularly, there are 3 types of recent graduates from CS or related fields: The ones that don't know crap, the ones that focus on these sort of problems, and the ones that are "generalists" because they dove into all sort of subjects during their degree.

I found out that the Jr people that excel at those sort of problems have a huge learning curve to climb to be productive in "production", real life environment. On the contrary, the "generalists" work better.

We stopped doing those sort of algorithm puzzles interviews after that realization, and we started getting really good Engineers with great real-life experience.

Re: Google outage – resolved

#865

Earlier quoted context omitted.

So you do think Leetcode is useful ? What's up with the flex ? Congratulations on the PhD and job, I guess ?. Looks like you could definitely invert a binary tree.

Leetcode is good for some things, but in my experience is it not a good factor to filter Software Engineers (it is a good metric to filter candidates to form a team for the IOI or similar). This is a response I wrote elsewhere: I've been growing Engineering teams for the last 10 years as hiring managers in different startups. At some point in our startup we had those kind of HackerRank questions as filters. The thing…

Great let me know when this scales to 100k people.

Re: Google outage – resolved

#866

Earlier quoted context omitted.

Completely agree with your analogy but have you ever seen any SLA that provides any additional liability? I haven't seen them - you are stuck either relying on hosting services SLA or DIY.

I cannot share details for the obvious reason, but yes - there are SLAs signed into contract directly behind the scenes which result in automatic payouts of a condition isn't met and it's not a simple credit. Enterprise level SLAs are crafted by lawyers in negotiations behind the scenes and are not the same as what you see on random public services. Our customers have them with us, and we have them with our vendors.…

That is a fair point. Is this a situation where you asymmetrically powerful? I have to imagine you would have considerable clout to represent a fair bit of their revenue in order to dictate terms. When I wrote my comment it was in the vein of a smaller organization.

Re: Google outage – resolved

#867
post #609

Earlier quoted context omitted.

The point is that you shouldn't put all your eggs in one basket. All services go down. If you're worried about someone else handling it when it goes down then host your own [1], otherwise you can use something different for each thing you need. Don't rely on Google for everything. 1- https://github.com/awesome-selfhosted/awesome-selfhosted#ema...

Yes, I know what's the point. But how do you avoid putting all your eggs in one basket? You can't host your email on more than one "provider" (including self host), and the vast majority of important services that you link your email to (bank, digital identity, taxes and other government services) does not allow you to have more than one linked to it; which means, that one goes down, you don't have one. Sure, I can g…

You shouldn't use you@company.com as your main email, you should have your own domain. So `something@yourdomain.com` will always be yours no matter if you self host or use 3rd party. I currently use Fastmail and i've been very happy with them. If they fail or turn evil, I'll switch to something else maintaining the same address. Emails themselves should be downloaded/backed-up regularly, kind of like important documents you'd have on your disk.

Re: Google outage – resolved

#868

Earlier quoted context omitted.

I cannot share details for the obvious reason, but yes - there are SLAs signed into contract directly behind the scenes which result in automatic payouts of a condition isn't met and it's not a simple credit. Enterprise level SLAs are crafted by lawyers in negotiations behind the scenes and are not the same as what you see on random public services. Our customers have them with us, and we have them with our vendors.…

That is a fair point. Is this a situation where you asymmetrically powerful? I have to imagine you would have considerable clout to represent a fair bit of their revenue in order to dictate terms. When I wrote my comment it was in the vein of a smaller organization.

I am but a technical cog in the machine my friend, while I know about what goes on in business and contract negoatiations I cannot comment on power dynamics. I would assume it's like any other negotiation - whomever has the greatest leverage has the power, I doubt it's ever fairly balanced.

Re: Google outage – resolved

#869
post #642

Earlier quoted context omitted.

Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.

colorblind (red/green) person here - 5% of the male population just don't see color enough for it to be an important visual clue. So sure, color-code your environments, but if you find someone about to do something to a red environment that they clearly should only be doing to a green environment, just check if they're seeing what you're seeing before you sack them ;)

My primary customer right now color codes both with "actual color", and with words - ie the RED environment is tagged with red color bars, and also big [black] letters in the [red] bars reading "RED"

Re: Google outage – resolved

#870
post #665
post #641

Earlier quoted context omitted.

If you alone were able to do it, then the system was designed badly. The bigger the impact, the more robust it has to be to prevent accidents.

The big mistake in the system is that everyone in the world is relying on Google services... These problems would have less impact with a more diverse ecosystem.

Just think if Google actually decided to "take the day off"

...like I did a dozen+ years ago: https://antipaucity.com/2008/01/09/what-if-google-took-the-d...

Post reply on HN