Google Is 2B Lines of Code, All in One Place • Hacker News Board

Re: Google Is 2B Lines of Code, All in One Place

#81

Post by dekhn » Wed, Sep 16, 2015, 7:24 PM UTC

I'm a google software engineer and it's nice to see this public article about our software control system. I think it has plusses and minuses, but one thing I'll say is that when you're in the coding flow, working on a single code base with thousands of engineers can be an intensely awesome experience.

Part of my job- although it's not listed as a responsibility- is updating a few key scientific python packages. When I do this, I get immediate feedback on which tests get broken and I fix those problems for other teams along side my upgrades. This sort of continuous integration has completely changed how I view modern software development and testing.

Re: Google Is 2B Lines of Code, All in One Place

#82

Post by malkia » Wed, Sep 16, 2015, 7:24 PM UTC

bigtunacan wrote: post #42

grouseway wrote: post #25

Earlier quoted context omitted.

Why not IBM? Why not Microsoft? Is that what the HN crowd has turned into? You now have to justify the non mainstream solution like a scared mega-corp career drone who will only go with the safest bet? Git isn't so awesome that it's inconceivable that people would be willing to use something else.

Personally; I wondered the same thing as the parent, why not Git? For me, this thought was not in a, "How could they possibly not choose Git!?", but more along the lines of, "There are a lot of smart people working at Google; I wonder what technical aspects of Mercurial might make it more suited to such a large code repository than Git?"

I'm at Google now, but would share my opinion of Perforce while I was at Treyarch - Perforce is simply very easy for anyone to use - from software engineer, to game designer, scripter, artist, producer, technical writer, linguist, translator, etc. You can announce your changes by simply saying - it's submitted in CL: 12345678 - grab it and you are done. Eases the communication.

Git and HG are not. Not that they can't be learned, but the learning curve is much higher.

Piper and CitC (mentioned in Rachel Potvik's videO) are even more advanced, I can work from my desktop machine, then go and continue from my laptop, or open and edit files directly from critique in an internal web-based editor.

Key to all this - make it accessible through the web! Anyone can work then from almost any machine. But at the same time the sources is accessible in the file system - even more awesome - so i can use emacs, and whatever other tools are there.

Re: Google Is 2B Lines of Code, All in One Place

#83

Post by georgemcbay » Wed, Sep 16, 2015, 7:28 PM UTC

lighthawk wrote: post #6

"The two internet giants (Google and Facebook) are working on an open source version control system that anyone can use to juggle code on a massive scale. It’s based on an existing system called Mercurial. “We’re attempting to see if we can scale Mercurial to the size of the Google repository,” Potvin says, indicating that Google is working hand-in-hand with programming guru Bryan O’Sullivan and others who help overs…

I've used git on repos far less than 2 billion lines of code and it was kind of a nightmare to clone and do basic local checkouts and normal everyday stuff. Out of the box, git's design is not well suited to large repos.

I know there are various workarounds for dealing with large repos but having had some experience using git on big-ish projects I can certainly understand some possible reasons why it wasn't their first choice given the size of their codebase.

Re: Google Is 2B Lines of Code, All in One Place

#84

Post by rictic » Wed, Sep 16, 2015, 7:29 PM UTC

727374 wrote: post #67

Really? This article sounds very over simplified, but I haven't worked at google so I wouldn't know. I'm assuming if you want to change some much depended on library, there's a way to up the version number so you don't hose all your downstream users. That's the way it worked at Amazon at least. Also, I wonder why the people in the story think Google's codebase is larger than that of other tech giants, not that it rea…

It's incumbent upon the person updating the library to get all users migrated to the new one. There are a few strategies for doing this though, including temporarily having two versions of the library.

There are also tools for making large scale changes safely and quickly.

Re: Google Is 2B Lines of Code, All in One Place

#85

Post by Splines » Wed, Sep 16, 2015, 7:30 PM UTC

a3n wrote: post #29

In the spirit of "You didn't build that," I wonder how many lines of code comprise the binaries that Google binaries run on? Windows, Linux, network stacks, Mercurial, etc, etc. I also wonder if there's a circular relationship anywhere in there.

It's turtles all the way down, and also includes all the hardware and people.

Re: Google Is 2B Lines of Code, All in One Place

#86

Post by kazinator » Wed, Sep 16, 2015, 7:31 PM UTC

I am unable to believe that Google has 2B lines of original code written from scratch at Google.

Maybe they are counting everything they use. Somewhere among those 2B lines is all the source code for Emacs, Bash, the Linux kernel, every single third-party lib used for any purpose, whether patched with Google modifications or not, every utility, and so on.

Maybe this is a "Google Search two billion" rather than a conventional, arithmetic two billion. You know, like when the Google engine tells you "there about 10,500,000 results (0.135 seconds)", but when you go through the entire list, it's confirmed to be just a few hundred.

Re: Google Is 2B Lines of Code, All in One Place

#87

Post by maximilianburke » Wed, Sep 16, 2015, 7:31 PM UTC

Lewisham wrote: post #28

rbinv wrote: post #2

Those are mind-boggling numbers. Although I kind of doubt that "almost every" engineer has access to the entire repo, especially when it comes to the search ranking stuff.

FWIW, apart from the previously mentioned sensitive stuff, we give engineering interns the same level of access we give full-time engineers. We keep things open because it makes things faster; we have an excellent code search tool that's great for navigating through the Piper repo (e.g. finding subclasses, finding uses of an API) which really speeds up dev time. When we're not talking about the sensitive stuff, there…

How are changes that affect sensitive code handled? Are the owners of that code on the hook for making any API updates that the person pushing the change can't make?

Re: Google Is 2B Lines of Code, All in One Place

#88

Post by nickpsecurity » Wed, Sep 16, 2015, 7:32 PM UTC

k33n wrote: post #4

Comparing "Google" to Windows isn't really a fair comparison. I'm sure all of the code that represents products that Microsoft has in the wild far exceeds 2B lines.

I also agree especially as this includes the whole Google software ecosystem and Microsoft has their own ecosystem. Microsoft's whole ecosystem of products that work together and run on Windows is much larger than Windows itself. Might not be as monolithically developed, though.

Re: Google Is 2B Lines of Code, All in One Place

#89

Post by durin42 » Wed, Sep 16, 2015, 7:35 PM UTC

sytse wrote: post #39

So a monolithic codebase makes it easier to make an organization wide change. Microservices make it easier to have people work and ship in independent teams. The interesting thing is that your can have have microservices with a monolithic codebase (as Google and Facebook are comprised of many services). But you can also have a monolithic service with many codebases (like our GitLab that uses 800+ gems that live in se…

Google has tons of services internally that talk via RPC. The monolithic repo means that it's much easier to hunt down and find people that are (say) using an outdated RPC method and help them fix their code.

(Just one example of how it's useful even when things are mostly services.)

Re: Google Is 2B Lines of Code, All in One Place

#90

Post by nevir » Wed, Sep 16, 2015, 7:36 PM UTC

dekhn wrote: post #81

I'm a google software engineer and it's nice to see this public article about our software control system. I think it has plusses and minuses, but one thing I'll say is that when you're in the coding flow, working on a single code base with thousands of engineers can be an intensely awesome experience. Part of my job- although it's not listed as a responsibility- is updating a few key scientific python packages. When…

Being able to make sweeping changes to a shared piece of code, and ensure that everyone's up to date (Hi, Rosie!) and not broken by your change (yay TAP train!) is phenomenal as well.