What does it take to make Google work at scale? • Hacker News Board

Re: What does it take to make Google work at scale?

#111

Post by jcrites » Tue, Aug 18, 2015, 8:51 AM UTC

faragon wrote: post #103

Wow. That's amazing. Anyone knows how they do a backup/restore of a GFS distributed file system? (in case they do backup of some sort)

I don't have any special knowledge of GFS, but one purpose of distributed file systems of that sort is generally to avoid the need for additional backup by replicating the data many times as part of its routine operation. The data storage layer itself replicates data many times, across multiple data centers, and continually verifies the replicas' integrity and re-replicates as needed.

The GFS paper describes replication in a bit more detail:

> Users can specify different replication levels for different parts of the file namespace. The default is three. The master clones existing replicas as needed to keep each chunk fully replicated as chunkservers go offline or detect corrupted replicas through checksum verification

Some other reasons you wouldn't backup a distributed file system (not saying there are no reasons ever [1]):

(1) Difficult to add another layer of backup without impacting performance unpredictably at backup time. It's more predictable to implement much of the replication synchronously within the request (while optionally some replicas to catch up out-of-band) (2) Files are differently important - some may warrant a greater degree of redundancy than others. The file system can understand this and take advantage of it; a separate backup system on top of the file system probably can't. (3) A standard backup/restore process often implies downtime during recovery. One goal of distributed systems is to avoid downtime by handling faults transparently. They continuously repair themselves. See: recovery-oriented computing. (4) A backup and restore process that's in any way intrusive on the operation of the system will not be easy to test on an ongoing basis the way that failure recovery will be tested constantly within the distributed file system. (In a big server fleets, drives will fail all the time, giving you no end of opportunities to exercise your recovery process.)

[1] One reason might be a defense against "unknown unknown" faults in the file system itself that cause it to irrecoverably lose track of data.

Re: What does it take to make Google work at scale?

#112

Post by tajano » Tue, Aug 18, 2015, 9:08 AM UTC

davis_m wrote: post #4

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

> "something as simple as keeping time"

Keeping time has been a difficult problem for... well, as long as we've tried keeping time. For an interesting historical account on past difficulties, you might check out the book "Longitude" [1].

[1] http://www.amazon.com/Longitude-Genius-Greatest-Scientific-P...

Re: What does it take to make Google work at scale?

#113

Post by badtoyx » Tue, Aug 18, 2015, 9:09 AM UTC

ms705 wrote: post #109

vayarajesh wrote: post #104

Is it possible to view the video or the presentation given in which the speaker talks more in detail explaining the slideS?

Hi, author here. The talk for which I put the slide deck together was given at a summer school and unfortunately not recorded, but if there is sufficient interest, I might tape a re-run and upload it. (Though unlikely to have time in the next month, so it might be a while.) In the meantime, the original papers (listed in the bibliographies at http://malteschwarzkopf.de/research/assets/google-stack.pdf and http://malt…

Please DO re-run the talk and share it with us. It would be quite valuable for a variety of audience. Thanks in advance!

Re: What does it take to make Google work at scale?

#114

Post by cronopios » Tue, Aug 18, 2015, 9:49 AM UTC

ms705 wrote: post #109

vayarajesh wrote: post #104

Is it possible to view the video or the presentation given in which the speaker talks more in detail explaining the slideS?

Hi, author here. The talk for which I put the slide deck together was given at a summer school and unfortunately not recorded, but if there is sufficient interest, I might tape a re-run and upload it. (Though unlikely to have time in the next month, so it might be a while.) In the meantime, the original papers (listed in the bibliographies at http://malteschwarzkopf.de/research/assets/google-stack.pdf and http://malt…

Wow, that would be awesome!

I hope you do it!

Re: What does it take to make Google work at scale?

#115

Post by mmoya » Tue, Aug 18, 2015, 10:08 AM UTC

davis_m wrote: post #4

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

> I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.

Check this talk: https://archive.fosdem.org/2015/schedule/event/ntimed_ntpd_r... . It's more difficult than it appear.

Re: What does it take to make Google work at scale?

#116

Post by xyzzyz » Tue, Aug 18, 2015, 11:43 AM UTC

faragon wrote: post #103

Wow. That's amazing. Anyone knows how they do a backup/restore of a GFS distributed file system? (in case they do backup of some sort)

Apart from other people mentioning that distributed file systems have replication as a part of the architecture, Google also uses tape[1][2].

[1] - http://www.tested.com/tech/1926-why-google-uses-tape-to-back... [2] - http://www.theregister.co.uk/2013/12/29/a_year_of_tape_tittl...

Re: What does it take to make Google work at scale?

#117

Post by xyzzyz » Tue, Aug 18, 2015, 11:48 AM UTC

yzh wrote: post #102

This may seem irrelevant, but as a GPU computing researcher, I'm still disappointed that GPU still hasn't made its way to be the first citizen at data centers. I know world of HPC looks much different, but I wonder when GPU can be vastly used in data centers, and will NVIDIA's Pascal architecture make a huge impact on this?

They are. With as much neural networks stuff as Google does, it's pretty obvious tons of GPU-hours must be in use.

http://www.wired.com/2013/05/gpus-in-the-data-center/

Re: What does it take to make Google work at scale?

#118

Post by i336_ » Tue, Aug 18, 2015, 1:01 PM UTC

jordz wrote: post #45

It's presentations like this that give me a that spark to go out and build crazy-shit. Sometimes you HAVE to reinvent the wheel because wheels just don't work to move objects the size of the planet the wheel is bound to. I'm not a fan of re-inventing the wheel by any means but it's this kind of stuff that makes me love going to work every day and basing ideas off presentations like this!

Well, it isn't every day you get to redefine π :D

Re: What does it take to make Google work at scale?

#119

Post by whitenoice » Tue, Aug 18, 2015, 2:30 PM UTC

ms705 wrote: post #109

vayarajesh wrote: post #104

Is it possible to view the video or the presentation given in which the speaker talks more in detail explaining the slideS?

Hi, author here. The talk for which I put the slide deck together was given at a summer school and unfortunately not recorded, but if there is sufficient interest, I might tape a re-run and upload it. (Though unlikely to have time in the next month, so it might be a while.) In the meantime, the original papers (listed in the bibliographies at http://malteschwarzkopf.de/research/assets/google-stack.pdf and http://malt…

Thank you for sharing this! and the consolidated papers list. Would be great if you can record and upload your next re-run!

Re: What does it take to make Google work at scale?

#120

Post by ska » Tue, Aug 18, 2015, 3:10 PM UTC

zo1 wrote: post #21

ska wrote: post #6

Earlier quoted context omitted.

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.

Sorry - missed this somehow.

People have mentioned distributed transactions and security, another area is synchronizing modeling with (hard or soft) real time inputs from separate hardware. There are a bunch of ways to get yourself tied up in knots once at least 2 physical bits of hardware are involved.