Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What does it take to make Google work at scale?

docs.google.com

Post reply on HN
Print view View original

31–40 of 134 posts

Re: What does it take to make Google work at scale?

#31

Post by pjungwir »

david_shaw wrote: post #18

> Wow, this file is really popular! Some tools might be unavailable until the crowd clears. Hopefully the irony isn't lost on the readers of this document :)

I think it has more to do with preventing concurrent writes stepping on each other. Google Docs must be the number one example of real-world systems doing Operational Transformation [1]. It's not like HN is bring down Google's servers.

Incidentally OT is another example of the value of in-sync clocks, for those asking about that elsewhere in this thread.

[1] https://en.wikipedia.org/wiki/Operational_transformation

Re: What does it take to make Google work at scale?

#32

Post by martincmartin »

davis_m wrote: post #4

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

A good article about this is "There is No Now" on ACM:

https://queue.acm.org/detail.cfm?id=2745385

Re: What does it take to make Google work at scale?

#33

Post by Filligree »

cbr wrote: post #25
untog wrote: post #19

Earlier quoted context omitted.

Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you. But like I said, we're all very different.

working on a giant campus that's isolated from the outside world You don't have to work in Mountain View! The remote offices are really nice, and are generally well integrated into their cities.

Right, even the one in San Francisco.

Re: What does it take to make Google work at scale?

#34

Post by lmorris84 »

zo1 wrote: post #21
ska wrote: post #6

Earlier quoted context omitted.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.

I believe Google Spanner obsesses over timestamps quite a lot to deliver distributed transactions with something called TrueTime [1].

[1] http://research.google.com/archive/spanner.html

Re: What does it take to make Google work at scale?

#35

Post by munin »

DannyBee wrote: post #24
david_shaw wrote: post #18

> Wow, this file is really popular! Some tools might be unavailable until the crowd clears. Hopefully the irony isn't lost on the readers of this document :)

What irony? To me, this isn't irony at all. It shows that google can't defeat the algorithmic complexity of things like consensus and shared state editing, and so gracefully degrades functionality. That shows a very good understanding of scaling - realizing any solution you choose has limits and tradeoffs, and thinking about and handling those limit cases sanely ahead-of-time, rather than waiting for it to fall over…

I got an error message saying the document wasn't available at all and to come back after the "crowd clears" which is much more ironic... after I refreshed a bunch it became available...

Re: What does it take to make Google work at scale?

#36

Post by patio11 »

nostrademons wrote: post #3
halotrope wrote: post #2

Kind of ironic: "Wow, this file is really popular! Some tools might be unavailable until the crowd clears"

Incidentally, that's part of scalability: knowing when you can drop functionality and the user won't care. For Google Docs, it's highly unlikely that a file with 1000s of viewers is actively being edited by 1000s of users, and more likely the link just got shared on the Internet. So you can drop edit functionality for everyone but the original author and they won't care. Another example - one that I've heard second-h…

Same at LinkedIn. (Source: presentation by their then-CEO at JavaOne, circa 2009.)

Re: What does it take to make Google work at scale?

#37

Post by davis_m »

munificent wrote: post #17
davis_m wrote: post #4

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

> Every review I hear about working at Google makes me want to stay away Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news . In general, it's a fantastic workplace.

Reading it back, my comment was harsher than I intended, especially with the reference to Amazon, but most of my aversion to working at a place like Google is just the shear size as well as where most of the offices are located.

I grew up in a tiny midwest town and I love it here. I would not enjoy living on either one of the coasts.

Many of my classmates in college couldn't wait to get out of the midwest. I have friends at Google, Amazon, Microsoft, and other large names in the tech industry, but more often than not, when I hear them talk about their jobs, even when they are talking about them positively, I am glad I stayed in the midwest because that is what fits me the best.

Re: What does it take to make Google work at scale?

#38

Post by mad44 »

ska wrote: post #6
davis_m wrote: post #4

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

Hybrid Logical Clocks offer a simple and feasible alternative to Google's custom hardware based TrueTime.

http://muratbuffalo.blogspot.com/2014/07/hybrid-logical-cloc...

Re: What does it take to make Google work at scale?

#39

Post by nostrademons »

untog wrote: post #30
DannyBee wrote: post #28

Earlier quoted context omitted.

"I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them)." This is true of almost everyone, of course. Every year i read the incoming intern abstracts, and they all literally say the same thing "I really would like to work on ". Literally all of them. That said, often you can work on them if you are good…

Are there ever new, fast-moving projects at Google? That's the core of my perception - I don't want to work on Google Docs, Gmail, etc. etc. or other large, established projects. I want to work on something small, iterate quickly, etc. - but to the outsider, I don't see any Google products doing that.

There are lots of them, but they tend to get canceled early. As an outsider, you only see projects once they've reached some baseline threshold of viability - a large number of projects never reach that, not because Google says they can't but just because they're bad ideas to begin with.

I spent about half my time at Google working on mundane improvements to search - visual redesigns, feature unification, infrastructure improvements - and half working on crazy green-field projects. Most of the crazy stuff was eventually canceled, and the stuff that did launch (eg. Google Authorship) ended up being a lot more toned-down than we initially envisioned. Ultimately I think I learned more from the crazy projects, but it's a very different kind of learning, much more experiential than factual.

The other thing you learn when you actually succeed at a crazy new idea is that people build up a tolerance to them really quickly. The first time we did an interactive doodle on the home page (PacMan...actually technically that was the second, but it was the first people noticed), everybody went wild, it was in all the newspapers, and we calculated people spent 4.82 million hours playing it. Now when an interactive doodle comes out, most people don't even notice. Remember that Google Docs, GMail, etc. were revolutionary in their day; it's only because they've become successful that you don't want to work on them.

Re: What does it take to make Google work at scale?

#40

Post by lg »

davis_m wrote: post #37
munificent wrote: post #17

Earlier quoted context omitted.

> Every review I hear about working at Google makes me want to stay away Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news . In general, it's a fantastic workplace.

Reading it back, my comment was harsher than I intended, especially with the reference to Amazon, but most of my aversion to working at a place like Google is just the shear size as well as where most of the offices are located. I grew up in a tiny midwest town and I love it here. I would not enjoy living on either one of the coasts. Many of my classmates in college couldn't wait to get out of the midwest. I have fri…

there's a chicago office, fwiw...
Post reply on HN