What does it take to make Google work at scale? • Hacker News Board

Re: What does it take to make Google work at scale?

#21

Post by zo1 » Mon, Aug 17, 2015, 7:44 PM UTC

ska wrote: post #6

davis_m wrote: post #4

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync?

And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.

Re: What does it take to make Google work at scale?

#22

Post by MattCruikshank » Mon, Aug 17, 2015, 7:44 PM UTC

tikhonj wrote: post #16

MattCruikshank wrote: post #8

Earlier quoted context omitted.

"Every review I hear about working at Google makes me want to stay away" I loved working there. Sure, who your boss is has a huge impact on your happiness. I ended up with a boss I enjoyed working for. Does that review make you want to stay away, too?

It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned . (Ideally I wouldn't want it impacting much at all , but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.

In my hiring process, I interviewed with multiple managers. I picked my favorite, and I ended up working for him. It didn't work out. Within a few weeks I was working for a new manager.

And frankly, I've never once worked for a company where I was immune from re-orgs.

Re: What does it take to make Google work at scale?

#23

Post by chetanahuja » Mon, Aug 17, 2015, 7:49 PM UTC

jasonpeacock wrote: post #7

ska wrote: post #6

Earlier quoted context omitted.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

i.e. A lot of distributed systems problems can be simplified if you can trust your clocks.

Or guarantee that two separate coordinates on space-time can have identical values... which would mean breaking some laws of physics.

Re: What does it take to make Google work at scale?

#24

Post by DannyBee » Mon, Aug 17, 2015, 7:50 PM UTC

david_shaw wrote: post #18

> Wow, this file is really popular! Some tools might be unavailable until the crowd clears. Hopefully the irony isn't lost on the readers of this document :)

What irony?

To me, this isn't irony at all. It shows that google can't defeat the algorithmic complexity of things like consensus and shared state editing, and so gracefully degrades functionality.

That shows a very good understanding of scaling - realizing any solution you choose has limits and tradeoffs, and thinking about and handling those limit cases sanely ahead-of-time, rather than waiting for it to fall over and hoping for the best.

Re: What does it take to make Google work at scale?

#25

Post by cbr » Mon, Aug 17, 2015, 7:51 PM UTC

untog wrote: post #19

munificent wrote: post #17

Earlier quoted context omitted.

> Every review I hear about working at Google makes me want to stay away Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news . In general, it's a fantastic workplace.

Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you. But like I said, we're all very different.

    working on a giant campus that's isolated from
    the outside world

You don't have to work in Mountain View! The remote offices are really nice, and are generally well integrated into their cities.

Re: What does it take to make Google work at scale?

#26

Post by pauleastlund » Mon, Aug 17, 2015, 7:51 PM UTC

tikhonj wrote: post #16

MattCruikshank wrote: post #8

Earlier quoted context omitted.

"Every review I hear about working at Google makes me want to stay away" I loved working there. Sure, who your boss is has a huge impact on your happiness. I ended up with a boss I enjoyed working for. Does that review make you want to stay away, too?

It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned . (Ideally I wouldn't want it impacting much at all , but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.

I spent about 5 years at Google starting in 2006. When I arrived I was assigned to one manager, but I wasn't _super_ excited about the project. A week in, another manager offered me another opportunity and got me permission to transfer. Problem solved.

A couple years later, a new hire got assigned to a team I was on. He was a little bummed because he'd really had his eye on another project. So we talked to our manager about it and he was allowed to transfer to the team he'd been hoping for.

I have more anecdotes like this, but the long and short of it is that in my experience Google is a lot less capricious and uncaring an organization than you imagine.

Re: What does it take to make Google work at scale?

#27

Post by matthewmacleod » Mon, Aug 17, 2015, 7:52 PM UTC

zo1 wrote: post #21

ska wrote: post #6

Earlier quoted context omitted.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.

Well, it's a reasonable solution to some distributed problems. I imagine that many distributed algorithms can be simplified a lot of you have a reliable and accurate time source. If you can build a reliable clock at less cost (development and/or overhead) than a time-insensitive algorithm would cost, then why not do it?

Re: What does it take to make Google work at scale?

#28

Post by DannyBee » Mon, Aug 17, 2015, 7:52 PM UTC

untog wrote: post #19

munificent wrote: post #17

Earlier quoted context omitted.

> Every review I hear about working at Google makes me want to stay away Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news . In general, it's a fantastic workplace.

Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you. But like I said, we're all very different.

"I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them)."

This is true of almost everyone, of course. Every year i read the incoming intern abstracts, and they all literally say the same thing "I really would like to work on ". Literally all of them.

That said, often you can work on them if you are good enough at what you do.

(But yes, often you have to prove that first, either internally or externally)

Re: What does it take to make Google work at scale?

#29

Post by pjungwir » Mon, Aug 17, 2015, 7:53 PM UTC

zo1 wrote: post #21

ska wrote: post #6

Earlier quoted context omitted.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.

Many security protocols use credentials that are only valid for a brief time. Here is a simple example of a problem caused by my server's time being more than 10 minutes off from the time on S3:

http://illuminatedcomputing.com/posts/2015/04/paperclip_expi...

Re: What does it take to make Google work at scale?

#30

Post by untog » Mon, Aug 17, 2015, 7:55 PM UTC

DannyBee wrote: post #28

untog wrote: post #19

Earlier quoted context omitted.

Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you. But like I said, we're all very different.

"I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them)." This is true of almost everyone, of course. Every year i read the incoming intern abstracts, and they all literally say the same thing "I really would like to work on ". Literally all of them. That said, often you can work on them if you are good…

Are there ever new, fast-moving projects at Google? That's the core of my perception - I don't want to work on Google Docs, Gmail, etc. etc. or other large, established projects. I want to work on something small, iterate quickly, etc. - but to the outsider, I don't see any Google products doing that.