The Friendship That Made Google Huge • Hacker News Board

Re: The Friendship That Made Google Huge

#61

Post by mav3rick » Tue, Dec 04, 2018, 5:05 AM UTC

continuations wrote: post #60

> The world’s most robust computer systems... used special hardware that could tolerate single bit-flips. But Google... bought cheaper computers that lacked that feature. > Together, Jeff and Sanjay wrote code to compensate for the offending machines. I'm guessing the article is referring to Google buying servers with non-ECC RAM. How do you write code to compensate for non-ECC RAM?

By pushing this complexity up the stack and also having redundancy in storage. What's not correct can be an error and result in retries from the client that may be routed to good storage.

Re: The Friendship That Made Google Huge

#62

Post by macintux » Tue, Dec 04, 2018, 5:12 AM UTC

jogjayr wrote: post #57

lowercased wrote: post #22

“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner. ======= Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.

Pivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level. I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses: 1. It seems like it would be more expensive - which might deter companies from thinking about it 2. If you're not already doing it, it's hard to…

I’ve paired a bit before, and #5 is definitely true. I can only handle 2-3 hours of it.

Re: The Friendship That Made Google Huge

#63

Post by continuations » Tue, Dec 04, 2018, 5:16 AM UTC

mav3rick wrote: post #61

continuations wrote: post #60

> The world’s most robust computer systems... used special hardware that could tolerate single bit-flips. But Google... bought cheaper computers that lacked that feature. > Together, Jeff and Sanjay wrote code to compensate for the offending machines. I'm guessing the article is referring to Google buying servers with non-ECC RAM. How do you write code to compensate for non-ECC RAM?

By pushing this complexity up the stack and also having redundancy in storage. What's not correct can be an error and result in retries from the client that may be routed to good storage.

But how do you even tell something's not correct? For example in the article they had to manually examine the index and find out some words were missing. How do you detect that not on the hardware level but up on the software level?

Re: The Friendship That Made Google Huge

#64

Post by habosa » Tue, Dec 04, 2018, 5:19 AM UTC

akhilcacharya wrote: post #53

sytelus wrote: post #51

Interesting tidbits: To solve problems at scale, paradoxically, you have to know the smallest details -Alan Eustace Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the…

> Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. I..don't think this is still accurate. New grad SWEs are L3's.

Yeah it's mostly accurate but drop level 1. Lowest I've seen for technical employee of any kind is 2. New grad is 3. 4 is about 2 years later. 5 is the point where you have "no expectation if advancement" and is the most common level for mid-career people.

Re: The Friendship That Made Google Huge

#65

Post by groby_b » Tue, Dec 04, 2018, 5:29 AM UTC

jogjayr wrote: post #57

lowercased wrote: post #22

“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner. ======= Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.

Pivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level. I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses: 1. It seems like it would be more expensive - which might deter companies from thinking about it 2. If you're not already doing it, it's hard to…

There've been studies on the "more expensive" part, and IIRC it's about a 15% premium in terms of development time - there've been a few studies. Here's one I could turn up quickly: https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia....

Simultaneously, you get a bunch of benefits. (Most importantly, there's no bus factor of 1 for anything)

Re: The Friendship That Made Google Huge

#66

Post by perfmode » Tue, Dec 04, 2018, 5:38 AM UTC

continuations wrote: post #63

mav3rick wrote: post #61

Earlier quoted context omitted.

By pushing this complexity up the stack and also having redundancy in storage. What's not correct can be an error and result in retries from the client that may be routed to good storage.

But how do you even tell something's not correct? For example in the article they had to manually examine the index and find out some words were missing. How do you detect that not on the hardware level but up on the software level?

Checksums

Re: The Friendship That Made Google Huge

#67

Post by dweekly » Tue, Dec 04, 2018, 5:50 AM UTC

joshuamorton wrote: post #44

stdplaceholder wrote: post #33

Earlier quoted context omitted.

Urs doesn't need a level on the individual contributor scale; he's a Senior Vice President.

Urs however does have one, he's listed as a Google fellow (L10) if you go looking. As a general comment, to compare, a director is L8 equivalent, and VP L9, (but on the management ladder) although at VP+ it gets tricky because there are subdivisions. Also worth noting that the management ladder goes up more. But the total number of people who are >L11 is probably <11, so it's not like there's any data on how exactly…

Director is L8, Sr Director is L9, VP is L10, SVP is L11.

Re: The Friendship That Made Google Huge

#68

Post by hn_throwaway_99 » Tue, Dec 04, 2018, 6:00 AM UTC

Wow, this article is pretty amazing to me. I have a joke (err, kind of a joke) that I like to say, "It's never a cosmic ray." What I mean by that is that when you are debugging and getting incredibly frustrated, right when you feel that you want to give up because you can't find the bug, that you shouldn't blame "cosmic rays" because 99.999% of the time it's a logic problem.

Here was a case where it actually was cosmic rays (at least, not a logic problem), and these guys found it. I also agree with the title and intro of the article - I think it's very possible that if these 2 exact people hadn't been at Google at the time than Google's path could have been very different. Makes me also appreciate how much luck is involved in any successful indeavor. Not luck that they found the bug (that was hard work), but luck that these 2 guys found themselves working at Google right when Google needed them.

Re: The Friendship That Made Google Huge

#69

Post by wpietri » Tue, Dec 04, 2018, 6:06 AM UTC

jogjayr wrote: post #57

lowercased wrote: post #22

“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner. ======= Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.

Pivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level. I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses: 1. It seems like it would be more expensive - which might deter companies from thinking about it 2. If you're not already doing it, it's hard to…

Regarding #5, you get used to it. It's intense at first, but after a while, it's just normal. One secret is to take actual breaks. And another is to not over-work. When pairing, there's none of that stuff that might stretch out your day. E.g., checking Hacker News. It's solid work time, and after 8 hours of that I'm definitely done for the day.

Re: The Friendship That Made Google Huge

#70

Post by nyc111 » Tue, Dec 04, 2018, 6:09 AM UTC

> On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row representing an indexed word. Sanjay pointed: a digit that should have been a 0 was a 1.

I was curious how he can see "a digit that should have been a 0 was a 1" in a "thick column of 1s and 0s"? Can you give a specific example. I'm not questioning that he did but I'd like to know a specific example.