Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google outage – resolved

news.ycombinator.com

Post reply on HN
Print view View original

821–830 of 870 posts

Re: Google outage – resolved

#821

Post by camjohnson26 »

awesomemonster wrote: post #806
FloatingUp wrote: post #725

Earlier quoted context omitted.

When I was working at a pretty big IT provider in the electronic banking sector, we (management and senior devs) made it an unspoken rule, that: - Juniors shall also handle production deployments regularly. - A senior person is always on call (even if only unofficially / off the clock). - Junior devs are never blamed for fuckups, irrespective of the damage they caused. That was the only way to help people develop rou…

Same thing -- used to work at a very large hosting provider. One of our big internal infra management teams wouldn't consider newhires fully "part of the team" until they had caused a significant outage. It was genuinely a right of passage, as one person put it, "to cause a measurable part of the internet to disappear". I got to see a lot of people pass through this right of passage, and it was always fun to watch. E…

Sounds like hazing.

Re: Google outage – resolved

#822

Post by robotnikman »

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

I've been in that situation before at one of my previous jobs, where some important IT infrastructure when down for the whole company. Nowhere as big of a scale as this, but it was easily one of the most stressful moments of my life

Re: Google outage – resolved

#823

Post by robotnikman »

zeven7 wrote: post #766
tarruda wrote: post #529

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

Several years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a small subset of queries for about an hour. As part of the postmortem I had to calculate how many people likely tried to interact with it while it was broken. It was a lot and really made me realize the magnitude of an otherwise seemingly small production failure.…

This sounds like a good practice and hopefully something they still do. Calculating the exact numbers would definitely help cement the experience and its consequences into your mind.

Re: Google outage – resolved

#824

Post by blaser-waffle »

rachelbythebay wrote: post #803
eric_khun wrote: post #669

If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...

Your nines or their nines? I bet if you personally can't use it, but their overall reliability meets the bar, then they're within SLA. Don't ask why I know this.

This person service-provides

Re: Google outage – resolved

#825

Post by vacheron »

xtracto wrote: post #790

On reddit thread there are some really good jokes about this [1] related to their asinine interview questions. Like: > Did they try to fix them by inverting a binary tree? >> Yeah maybe implementing a quick LRU cache on the nearest whiteboard will help them out here >> Did they try checking what shape their manhole cover is? >> Dev ops was too busy out counting all the street lights in the United States [1] https://w…

AFAIK (as someone who does not work at Google), "brain teaser" questions have not been asked at Google for almost a decade.

Every time I see complaints like this I can't help but think the posters have a chip on their shoulder from being rejected by Google or a similarly selective company. It's never a commentary on any fundamental issues with these types of interview questions and no one ever comes up with a better process that is equally scaleable and effective for hiring generalists. In my experience interviewing with dozens of small, medium and large companies, the vast majority of technical roles require these types of questions now. These come off as criticisms against Google specifically for asking hard variants of these questions.

I don't personally have any issue with companies asking questions like these as long as they don't simply look for "a correct and optimal solution coded up perfectly whilst under stress in under 30 minutes", but rather the process of solving the problem.

Re: Google outage – resolved

#826

Post by vacheron »

xtracto wrote: post #808
mav3rick wrote: post #807

Earlier quoted context omitted.

None of these are asked in G interviews. The commenters are asinine.

No? https://twitter.com/mxcl/status/608682016205344768?lang=en

This story seems like a bad example of a false negative. There were claims by Google insiders that Max was given a particularly easy interview as a formality. I am by no means a brilliant programmer and also rusty with algorithms, but given the structure and question, I would be surprised if most people couldn't figure out how to invert a binary tree within a matter of minutes.

Max himself later opened up[0] and admitted to being difficult to work with. It is entirely likely that he was rejected based on his personality and not his ability. As someone who contributed to Homebrew many years ago I would not be surprised if this was the case. In their own words: "I am often a dick, I am often difficult, I often don’t know computer science". I am not sure why any company would want to hire someone like that and put the culture of the team in jeopardy.

[0] - https://www.quora.com/Whats-the-logic-behind-Google-rejectin...

Re: Google outage – resolved

#827

Post by BugWatch »

tux3 wrote: post #183
have_faith wrote: post #56

Gmail said my account was "temporarily" unavailable... had a moment considering if it wasn't temporary. Good reminder to remove my reliance on gmail especially.

Thunderbird prompting me to login, and getting "Google does not recognize this email address" as a reply was a nice adrenaline spike, until I checked the status on HN!

I am still experiencing this problem on all my synced-to-Thunderbird Gmail accounts, so it either hasn't been completely fixed yet, or there's another ongoing issue.

Re: Google outage – resolved

#828

Post by tasuki »

alias_neo wrote: post #701
tasuki wrote: post #682

Earlier quoted context omitted.

Why not? (I think I can find some cases where production access from an engineer's workstation is a good idea)

It can be efficient, particularly in smaller companies, but that's where exactly this rule should be applied. In some industries, security and customer requirements will at times mandate that developer workstations have no access to production. Deployments must even be carried out using different accounts than those used to access internal services, for security and auditing purposes. There are of course good reasons…

Congratulations - you found a counterexample yourself: engineers in small companies.

Re: Google outage – resolved

#829

Post by ehsankia »

eric_khun wrote: post #669

If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...

It seems like they claim around 45m

https://techcrunch.com/2020/12/14/gmail-youtube-google-docs-...

Re: Google outage – resolved

#830

Post by GreenWatermelon »

papito wrote: post #734
zentiggr wrote: post #721

Earlier quoted context omitted.

When I meet the engineer who can design for the unknown unknowns, I will bow to them.

The trick is to be paranoid. You literally sit down and think exclusively about what COULD go wrong.

Anxiety is a bitch.
Post reply on HN