Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Is 2B Lines of Code, All in One Place

wired.com

Post reply on HN
Print view View original

31–40 of 347 posts

Re: Google Is 2B Lines of Code, All in One Place

#31

Post by jpollock »

jfkw wrote: post #27

How do the monolithic repository companies handle dependencies on external source code? Are libraries and large projects e.g. RDBMS generally vendored/forked into the monolithic repositories, regardless of whether the initial intent is to make significant changes?

There's typically a subdirectory called third_party, with subdirectories for each vendor, product and version. If the team is smart, they will also enact a rule saying "only one version". If you're really, really smart, local changes are kept as a set of patches, keeping them separate from the imported tar file.

So, for source deliveries:

  third_party/apache/httpd/2.4/release.tgz
                              /patch.tgz
                              /Makefile (or other config)
  third_party/apache/httpd/2.2/release.tgz
                             /patch.tgz                             
  ...

Re: Google Is 2B Lines of Code, All in One Place

#32

Post by packetslave »

rbinv wrote: post #2

Those are mind-boggling numbers. Although I kind of doubt that "almost every" engineer has access to the entire repo, especially when it comes to the search ranking stuff.

I'm a random engineer and I have access to probably 99% of those 2B lines of code. The restricted stuff is a pretty tiny subset of the whole.

Re: Google Is 2B Lines of Code, All in One Place

#33

Post by comrh »

ChuckMcM wrote: post #17

I will say that I saw and experienced many things that changed my definition of 'large' at Google, but the most amazing was the source code control / code review / build system that kept it all together. The bad news was that it allowed people to say "I've just changed the API to to support the initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the…

> With the orb on their desk telling them at that their integration and unit tests were passing.

Anymore info on this? Is it in house hardware?

Re: Google Is 2B Lines of Code, All in One Place

#35

Post by cmrdporcupine »

urda wrote: post #13
lighthawk wrote: post #6

"The two internet giants (Google and Facebook) are working on an open source version control system that anyone can use to juggle code on a massive scale. It’s based on an existing system called Mercurial. “We’re attempting to see if we can scale Mercurial to the size of the Google repository,” Potvin says, indicating that Google is working hand-in-hand with programming guru Bryan O’Sullivan and others who help overs…

Because Google and Facebook are using Mercurial over Git internally. Edit: And for those that are just shocked that git isn't the answer. Facebook: https://code.facebook.com/posts/218678814984400/scaling-merc... Google: http://www.primordia.com/blog/2010/01/23/why-google-uses-mer...

I don't think I'm leaking anything by saying that that article about Google is incorrect. Mercurial isn't used at Google, not anywhere I've seen. Probably that article has something to do with the now-defunct Google Code site, which did in fact use Mercurial.

Re: Google Is 2B Lines of Code, All in One Place

#36

Post by kuschku »

thomashabets2 wrote: post #19
kuschku wrote: post #10

This explains quite some things. Still, this is not a very forward-thinking solution. Building and combining microservices – effectively UNIX philosophy applied to the web – is the most effective way to make progress. EDIT: Seems like I misunderstood the article – from the way I read it, it sounded like Google has a monolithic codebase, with heavily dependent products, deployed monolithically. As zaphar mentioned, it…

That's why Google is so unsuccessful at scaling technical solutions, unlike you they're not forward-thinking.

No, it’s not that they are unsuccessful, it’s that they are unable to maintain it properly. Already today they have tons of open security issues.

Or think about April 1st, when they set a Access-Control-Location: * header on google.com because someone wrote the com.google easteregg.

Read the post from the SoundCloud dude from yesterday to find out how to do software management properly (hint: modularization is everything)

Re: Google Is 2B Lines of Code, All in One Place

#37

Post by petra »

Lewisham wrote: post #28
rbinv wrote: post #2

Those are mind-boggling numbers. Although I kind of doubt that "almost every" engineer has access to the entire repo, especially when it comes to the search ranking stuff.

FWIW, apart from the previously mentioned sensitive stuff, we give engineering interns the same level of access we give full-time engineers. We keep things open because it makes things faster; we have an excellent code search tool that's great for navigating through the Piper repo (e.g. finding subclasses, finding uses of an API) which really speeds up dev time. When we're not talking about the sensitive stuff, there…

I though Google's search advantage really speeds up software development. I wonder if they use the same principle in alphabet across other engineering disciplines - creating some unique knowledge tools, and how do they look.

Re: Google Is 2B Lines of Code, All in One Place

#38

Post by jordigh »

cmrdporcupine wrote: post #35
urda wrote: post #13

Earlier quoted context omitted.

Because Google and Facebook are using Mercurial over Git internally. Edit: And for those that are just shocked that git isn't the answer. Facebook: https://code.facebook.com/posts/218678814984400/scaling-merc... Google: http://www.primordia.com/blog/2010/01/23/why-google-uses-mer...

I don't think I'm leaking anything by saying that that article about Google is incorrect. Mercurial isn't used at Google, not anywhere I've seen. Probably that article has something to do with the now-defunct Google Code site, which did in fact use Mercurial.

Google devs are working (with some help from Facebook) on getting hg into Google, but they're not there yet. Look at all the work that Google has been putting into hg. It's not purely out of the goodness of their hearts:

https://selenic.com/hg/log?rev=@google.com&revcount=200

I always see a lot of Google and Facebook at the Mercurial sprints.

Re: Google Is 2B Lines of Code, All in One Place

#39

Post by sytse »

So a monolithic codebase makes it easier to make an organization wide change. Microservices make it easier to have people work and ship in independent teams. The interesting thing is that your can have have microservices with a monolithic codebase (as Google and Facebook are comprised of many services). But you can also have a monolithic service with many codebases (like our GitLab that uses 800+ gems that live in separate codebases). And of course you can have a monolithic codebase with a monolithic service (a simple php app). And you can have microservices with diverse codebases (like all the hipsters are doing).

I'm wondering if microservices force you to coordinate via the codebase just like using many codebases force you to coordinate via the monolithic service. Does the coordination has to happen somewhere? I wonder if early adopters of microservices in many codebases (SoundCloud) are experiencing coordination problems trying to change services.

Re: Google Is 2B Lines of Code, All in One Place

#40

Post by sytse »

So a monolithic codebase makes it easier to make an organization wide change. Microservices make it easier to have people work and ship in independent teams. The interesting thing is that your can have have microservices with a monolithic codebase (as Google and Facebook are comprised of many services). But you can also have a monolithic service with many codebases (like our GitLab that uses 800+ gems that live in separate codebases). And of course you can have a monolithic codebase with a monolithic service (a simple php app). And you can have microservices with diverse codebases (like all the hipsters are doing).

I'm wondering if microservices force you to coordinate via the codebase just like using many codebases force you to coordinate via the monolithic service. Does the coordination has to happen somewhere? I wonder if early adopters of microservices in many codebases (SoundCloud) are experiencing coordination problems trying to change services.

Post reply on HN