Live data from Hacker News

Google Is 2B Lines of Code, All in One Place

wired.com

231–240 of 347 posts

Re: Google Is 2B Lines of Code, All in One Place

#231

Earlier quoted context omitted.

I was at Google for a long time and it was almost entirely Perforce (some Googlers used git) until it was Piper (which is very similar to Perforce). L&S used CVS at Stanford for the original Backrub/Google code and for some time after the establishment of Google, Inc.

Is Piper a wrapper around Perforce or built from scratch? What is it exactly, if you don't mind me asking?

It was built from scratch because it was designed as a replacement for Perforce because Perforce is expensive and the G used wrappers around it, anyways.

The commands are still basically the same as Google's wrappers around Perforce and the learning curve was non-existent for someone who was used to using Perforce at Google.

Re: Google Is 2B Lines of Code, All in One Place

#232
post #92

I will say that I saw and experienced many things that changed my definition of 'large' at Google, but the most amazing was the source code control / code review / build system that kept it all together. The bad news was that it allowed people to say "I've just changed the API to to support the initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the…

Can you (or somebody) explain how that works in practice? Here I am ready to deploy some new feature to gmail. In the meantime, I'm getting a steady stream of API changes. Can I build and release gmail to a specific revision number, and only incorporate the changes when I am ready, or are all release essentially off of the tip of the tree. I don't need specifics, just the general idea. Where we work basically every p…

In theory: Everyone works from head, so there is only one version of everything.

When you write a feature, you follow these steps.

1. Write code and submit to the main (and only) branch. Hide your feature behind a flag. 2. Releases happen from head (with no regard to your feature) 3. When you want to enable your feature, you flip a flag to turn it on.

In practice: Things are messier.

Re: Google Is 2B Lines of Code, All in One Place

#233
post #92

I will say that I saw and experienced many things that changed my definition of 'large' at Google, but the most amazing was the source code control / code review / build system that kept it all together. The bad news was that it allowed people to say "I've just changed the API to to support the initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the…

Can you (or somebody) explain how that works in practice? Here I am ready to deploy some new feature to gmail. In the meantime, I'm getting a steady stream of API changes. Can I build and release gmail to a specific revision number, and only incorporate the changes when I am ready, or are all release essentially off of the tip of the tree. I don't need specifics, just the general idea. Where we work basically every p…

Yes, you sync you workspace to a particular revision and work within that. Syncing is a manually triggered process. And generally speaking, people changing APIs are required to find and fix all users (which isn't that hard with the infrastructure available). There is still some churn created by the constant stream of changes, but it's usually not on the changing APIs level.

Re: Google Is 2B Lines of Code, All in One Place

#234
post #172
post #81

I'm a google software engineer and it's nice to see this public article about our software control system. I think it has plusses and minuses, but one thing I'll say is that when you're in the coding flow, working on a single code base with thousands of engineers can be an intensely awesome experience. Part of my job- although it's not listed as a responsibility- is updating a few key scientific python packages. When…

If I recoup with latest news: The ranking algorithm is in one repo, Chrome and Android in another, and all the rest is in Piper? What is all the rest? Gmail, Calendar, the defunct Reader, the car, Maps, and OnHub... How does that fit with Alphabet? Is Piper a cross-functional initiative of Alphabet now?

The ranking algorithm (and other restricted stuff) is in Piper, but locked down as to who can browse it, check it out, etc. There's some magic in the build system that lets you compile/link against it without needing to see it.

Re: Google Is 2B Lines of Code, All in One Place

#236
post #117
post #95

Earlier quoted context omitted.

How is this a side effect of it being in the same repository/

Say I make a change to a commonly used library (let's say deprecating a function, and replacing it with another): * I can see literally _every use_ of the old function. * I can run the tests for everyone who uses that function. * * this is automated; the build/test tooling can figure out the transitive set of build/test targets that are affected by such a change. * I can (relatively) easily update _every use_ of the…

We've been really crappy at marketing this but we have a multiple repos answer that is semantically close to identical to a monolithic repo. Any work flow or command line works identically in both (there are a few corner cases that don't work but almost 100% of the stuff works like you'd expect).

It's sort of like a partial clone, you can clone any subset you want, work on it, add in other parts, the tool takes care of making sure the stuff you add in is lined up (if you cloned a week ago and you add in another repo, it's rolled back so it matches time/commit wise).

If you want to search for something it's the same command in a collection of repos as it is in a monolithic repo:

        bk -U grep refactor_me
but truth in advertising, that searches only what you have populated. The google answer is "ship grep to the data" so they'll search everything. Google wins if you have their datacenter. We win on UI, it's the same everywhere.

Our design is much more tightly coupled than Git's submodules. We manage the subrepos so that they are in sync just like files are in sync. What I mean by that is if you have two files modified in the same commit, you can't pull that commit and only get one of the files updated, they are both updated because that's what happened in that commit. We've provided the same semantics for collections of repositories. Git doesn't, getting those semantics is an exercise left to the user.

We get better performance because you can clone as little as one repository, what we call the product. It would be easy and fun to put a Unix distro in this system and have the top repo just have a makefile and you type make X11 and it populates the compiler, libc, the X11 sources, and builds them.

It's commercial so maybe that's uninteresting but if you want submodules that work and scale in performance, we might be worth a look (sorry for the marketing, if that's against the rules here then I'll learn how to delete a post).

Read more here, comments welcome, it's beta copy:

http://www.bitkeeper.com/nested

Re: Google Is 2B Lines of Code, All in One Place

#238
post #2

Those are mind-boggling numbers. Although I kind of doubt that "almost every" engineer has access to the entire repo, especially when it comes to the search ranking stuff.

Why wouldn't almost every engineer have access to almost the entire repo? Most of Google's code is only relevant to another company working at Google scale, such as Facebook, Microsoft, Apple, etc. These are the companies with deep pockets that are willing to spend lots of money to acquire technology that will help them compete with Google. But none of these companies will buy code that's been stolen from Google.

The most famous corporate trade secret, the Coke formula, was stolen by two employees who attempted to sell it to Pepsi. Pepsi alerted Coke, the companies worked together to bring in the FBI, and both employees went to prison: http://www.cnn.com/2007/LAW/05/23/coca.cola.sentencing/

Re: Google Is 2B Lines of Code, All in One Place

#239

Earlier quoted context omitted.

I'm still trying to figure out why having everything dumped in one big pile is something worth bragging about. I'd far rather have code sorted well into proper repositories.

So much is shared, though, right? Which is why Android is sorted into proper repositories but still has the 'repo' front-end wrapper to make sure you're getting the right versions of everything you need. If I wanted to change something fundamental, like I found a 10% speedup in Protobuf wire decode by changing the message slightly, there are likely very many services that all need it. Everyone at Google operates on H…

From my perspective, at least, this design seems to explain why Google websites are so frequently broken in the ways different services integrate. Because Googlers edit shared resources that affect products they don't personally work on, and they just trust automated tests, which almost certainly miss a lot of the edge cases I encounter.

I admit that I'm not an expert at large software development, but this seems to nearly fully explain Google's declining code quality.

Re: Google Is 2B Lines of Code, All in One Place

#240
post #109

Just because people are talking about it: I work at MSFT, and the numbers Wired quotes for the lines of code in Windows are not even close to being correct. Not even in the same order of magnitude. Their source claims that Windows XP has ~45 million lines of code. But that was 14 years ago. The last time Windows was even in the same order of magnitude as 50 million LOC was in the Windows Vista timeframe. EDIT: And, r…

I assume they don't use msbuild, because they don't completely hate themselves.
Post reply on HN