Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Is 2B Lines of Code, All in One Place

wired.com

Post reply on HN
Print view View original

61–70 of 347 posts

Re: Google Is 2B Lines of Code, All in One Place

#61

Post by spectral321 »

lighthawk wrote: post #59
grouseway wrote: post #25

Earlier quoted context omitted.

Why not IBM? Why not Microsoft? Is that what the HN crowd has turned into? You now have to justify the non mainstream solution like a scared mega-corp career drone who will only go with the safest bet? Git isn't so awesome that it's inconceivable that people would be willing to use something else.

> Git isn't so awesome that it's inconceivable that people would be willing to use something else. Inconceivable? No, I just asked why. Git to me is a great solution for source control, so I want to understand it's deficiency for handling large amounts of code, especially considering it is used for managing Linux source and handles everything in GitHub I've used quite well.

Git is fine at scaling horizontally (many repositories), so things like GitHub work well. Neither Git nor Mercurial do well at scaling to a single massive repository; getting Mercurial to do so was easier because we can implement it as out-of-core extensions (see http://bitbucket.org/Google/narrowhg, for example).

There's also internal-only things to deal with, such as the CitC integration. Because of some of the design decisions behind CitC, storing a typical .git or a .hg directory in CitC is essentially impossible. Mercurial's .hg directory is intentionally a black-box - you interact with the repo using the hg tool. The .git directory can be seen as an API - there are at least three implementations that matter for our purposes, and if we change something in the .git directory, we lose the editor/IDE integration powered by the ones we didn't fix (or we did fix, but they haven't been released yet; some of those are linked into products that have a commercial release cycle).

Re: Google Is 2B Lines of Code, All in One Place

#62

Post by Filligree »

kuschku wrote: post #58
zaphar wrote: post #53

Earlier quoted context omitted.

Google runs practically everything internally as services. Nothing about the code repository makes it impossible to run microservices. Where did you get the idea that google runs a single monolithic app for everything?

The article claimed the code, and they way it's run, is a monolithically developed and deployed product. If that’s not the case, I apologize for misunderstanding it. But if it was the case, I wanted to state that it might not be wise, for the same reasons as this thread mentioned https://news.ycombinator.com/item?id=10195423 EDIT: Thanks for telling me, though! Always nice to be proven wrong, as at least I learnt som…

It's not the case.

You should think of Piper as a single filesystem which permits atomic multi-file edits. And that's about it; there's nothing in that which forces any particular release structure on you.

Re: Google Is 2B Lines of Code, All in One Place

#63

Post by spectral321 »

kuschku wrote: post #58
zaphar wrote: post #53

Earlier quoted context omitted.

Google runs practically everything internally as services. Nothing about the code repository makes it impossible to run microservices. Where did you get the idea that google runs a single monolithic app for everything?

The article claimed the code, and they way it's run, is a monolithically developed and deployed product. If that’s not the case, I apologize for misunderstanding it. But if it was the case, I wanted to state that it might not be wise, for the same reasons as this thread mentioned https://news.ycombinator.com/item?id=10195423 EDIT: Thanks for telling me, though! Always nice to be proven wrong, as at least I learnt som…

The talk did mention (briefly) that monolithic codebase and monolithic binaries/software aren't strictly related. It's likely that monolithic software is easier in a monolithic codebase, but I don't think that microservices are harder in a monolithic codebase. Yes, we tend to statically link, but that's for library dependencies (i.e. things like the protocol buffers libraries/definitions). I don't work at all in this area, so this is a guess, but I imagine that it's extremely rare for unrelated teams to have application logic linked into the same binary - just making a release would involve coordination from so many teams :)

Re: Google Is 2B Lines of Code, All in One Place

#64

Post by sytse »

sytse wrote: post #39

So a monolithic codebase makes it easier to make an organization wide change. Microservices make it easier to have people work and ship in independent teams. The interesting thing is that your can have have microservices with a monolithic codebase (as Google and Facebook are comprised of many services). But you can also have a monolithic service with many codebases (like our GitLab that uses 800+ gems that live in se…

I think that advantages mentioned in the presentation about the monolithic codebase can be achieved if you have one source code server that is open to everyone (for example GitLab with most projects set to internal). Some of the tools will be easier to write for one repository than iterating over many, but that seems solvable.

The biggest advantage seems to be that when you are an author of a dependency you can propose upgrades to all services that use your application. It is not clear to me but it seems that for small changes you can just force that change on the code owners. This ensures that the dependency author incurs the cost of a change (as is done for API changes in the Linux kernel) and that you do not need to version the API of the dependency.

Interestingly Google recently started marking API's private by default. So they are moving in the direction of explicit API management.

As soon as you work with people that are outside your control (as is common in open source) you would need to version the API as well in my opinion.

Re: Google Is 2B Lines of Code, All in One Place

#65

Post by sandGorgon »

What are the best practices to follow in a single-repo-multiple-projecrs world? Some people recommend git submodule, others recommend subtree.

How do you guys manage alerts and messages - does every developer get a commit notification,or is there a way to filter out messages based upon submodule.

How does branching and merging work?

I'm wondering what processes are used by non-Google/FB teams to help them be more productive in a monolithic repo world.

Re: Google Is 2B Lines of Code, All in One Place

#66

Post by ajross »

sandGorgon wrote: post #65

What are the best practices to follow in a single-repo-multiple-projecrs world? Some people recommend git submodule, others recommend subtree. How do you guys manage alerts and messages - does every developer get a commit notification,or is there a way to filter out messages based upon submodule. How does branching and merging work? I'm wondering what processes are used by non-Google/FB teams to help them be more pro…

FWIW: git submodules are not a single repo by definition. It's just a way to automate the checkout of specifically-versioned external projects without requiring hackery like packing tarballs into the project source. It has its uses, but it's definitely not what they're talking about here.

Re: Google Is 2B Lines of Code, All in One Place

#67

Post by 727374 »

Really? This article sounds very over simplified, but I haven't worked at google so I wouldn't know. I'm assuming if you want to change some much depended on library, there's a way to up the version number so you don't hose all your downstream users. That's the way it worked at Amazon at least. Also, I wonder why the people in the story think Google's codebase is larger than that of other tech giants, not that it really matters.

Re: Google Is 2B Lines of Code, All in One Place

#68

Post by stock_toaster »

devit wrote: post #22
ChuckMcM wrote: post #17

I will say that I saw and experienced many things that changed my definition of 'large' at Google, but the most amazing was the source code control / code review / build system that kept it all together. The bad news was that it allowed people to say "I've just changed the API to to support the initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the…

The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted. The Linux kernel generally uses this policy for internal APIs for example.

  > The solution to the excessive API change problem is to force
  > whoever changes the API to fix all the consumers himself
  > before the change is accepted.
This doesn't seem scalable. Let's consider the case of one api endpoint being changed by one developer, to add a new param to a function call. Further assume that this impacts hundreds of projects.

Does it really make sense to make one developer update those hundred projects? Not only will it take forever for it to get finished (possibly never if there are new consumers of this api coming online frequently), but the developer of the core api may not have any experience in the impacted consumers of this codebase. I think the end result of this policy would be nothing once written ever would get updated, and new apis would just be added all the time (api explosion).

Re: Google Is 2B Lines of Code, All in One Place

#69

Post by teraflop »

revelation wrote: post #26
devit wrote: post #22

Earlier quoted context omitted.

The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted. The Linux kernel generally uses this policy for internal APIs for example.

I guess that works for the Linux kernel, but I would presume that for a large distributed operation like Google it would be much better to simply deprecate/version APIs and have the project teams update to a deadline. I mean, it's presumably impossible to have a single computer running a single OS build all of the Google software and run the testing.

Google is rumored to have an extremely powerful distributed compilation farm. It wouldn't surprise me if a single developer could make a change that affects the entire codebase and test it themselves.

Re: Google Is 2B Lines of Code, All in One Place

#70

Post by yongjik »

One humorous side-effect of having all that code viewable (and searchable!) by everyone was that the codebase will contain whatever typo, error, or mistake you can think of (and convert into a regular expression).

I remember seeing an internal page with dozens of links for humorous searches like "interger", "funciton", or "([A-Z][a-z]+){7,} lang:java"...

Post reply on HN