Google Is 2B Lines of Code, All in One Place • Hacker News Board

Re: Google Is 2B Lines of Code, All in One Place

#261

Post by thrownaway2424 » Thu, Sep 17, 2015, 3:29 AM UTC

packetslave wrote: post #234

tajen wrote: post #172

Earlier quoted context omitted.

If I recoup with latest news: The ranking algorithm is in one repo, Chrome and Android in another, and all the rest is in Piper? What is all the rest? Gmail, Calendar, the defunct Reader, the car, Maps, and OnHub... How does that fit with Alphabet? Is Piper a cross-functional initiative of Alphabet now?

The ranking algorithm (and other restricted stuff) is in Piper, but locked down as to who can browse it, check it out, etc. There's some magic in the build system that lets you compile/link against it without needing to see it.

There is considerable magic of that kind, actually, because the build system has to be able to prevent insufficiently-entitled engineers from being able to back out the protected source code, while still allowing the program depending on the protected code to be built, linked, and debugged.

Re: Google Is 2B Lines of Code, All in One Place

#262

Post by thrownaway2424 » Thu, Sep 17, 2015, 3:34 AM UTC

stock_toaster wrote: post #68

devit wrote: post #22

Earlier quoted context omitted.

The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted. The Linux kernel generally uses this policy for internal APIs for example.

> The solution to the excessive API change problem is to force > whoever changes the API to fix all the consumers himself > before the change is accepted. This doesn't seem scalable. Let's consider the case of one api endpoint being changed by one developer, to add a new param to a function call. Further assume that this impacts hundreds of projects. Does it really make sense to make one developer update those hundre…

It maybe isn't scalable, but that's part of the benefit. If you want to make a change to a widely-used API, it's going to be a lot of work, and it's not going to be a lot of work for the users of the API, it's going to be a lot of work for _you_ because _you_ are required to do it yourself. This prevents a lot of API churn unless the benefit is clear and sufficiently large.

If it was any other way you'd rapidly reach a useless equilibrium where random engineers were demanding that thousands of other engineers fulfill unfunded mandates for what might turn out to be negligible benefits.

Re: Google Is 2B Lines of Code, All in One Place

#263

Post by dblotsky » Thu, Sep 17, 2015, 3:39 AM UTC

Even if the numbers are off, the assumption that 40M lines of code take less effort to write than 2B lines of code commits the fallacy that effort is proportional to number of lines of code. Come on, Wired, you can do better.

Re: Google Is 2B Lines of Code, All in One Place

#264

Post by thrownaway2424 » Thu, Sep 17, 2015, 3:42 AM UTC

breatheoften wrote: post #248

Are the source of piper and the build tools also in the mono repo and also developed/deployed off the head branch? Seems like a random engineer could royally fubar things if they broke a service which the build system depends on ...

You said "developed/deployed" as if it were the same thing. Even if you somehow checked in the giant flaw, bypassing all code review and automated testing, it's not like that would suddenly appear in production. Google isn't some PHP hack where you just copy a tarball to The Server. Binaries of even slightly important systems typically go through many stages of deployment, first into unimportant test systems, then usually very, very slowly into production with lots of instrumentation and of course, quick and easy methods of rolling back to the previous release.

Re: Google Is 2B Lines of Code, All in One Place

#265

Post by archimedespi » Thu, Sep 17, 2015, 3:44 AM UTC

ksk wrote: post #208

buffoon wrote: post #194

Earlier quoted context omitted.

It's really not that fast. The filesystem is a total dog (MFT contention) to the point that manipulating lots of small file is up to two orders of magnitude slower than ext4. This is made bearable thanks to SSDs being on the market. Also the amount of friction getting stuff built and running and maintaining it is detrimental to general productivity meaning you piss execution time out of the window regularly just fixi…

I can't say I'm surprised to see people eager to point out how Windows sucks. And sure, maybe it does. However, the fundamental point you're missing is that I don't think that Windows was ever positioned as this OS that was designed for every single type of workload out there (not withstanding marketing noise). Windows is a very general purpose OS meant for general purpose 'mainstream' things. Things that hundreds of…

At least you don't ever really need to defragment ext4, unlike NTFS.

Re: Google Is 2B Lines of Code, All in One Place

#266

Post by xur17 » Thu, Sep 17, 2015, 3:48 AM UTC

temuze wrote: post #176

Assuming these numbers are right... (15 million lines of code changed a week) / (25,000 engineers) = 600 LOC per engineer per week Is ~120 LOC per engineer per workday normal at other companies?

It really depends on what you're writing. Lower level c / c++, doubtful. Python, javascript, java, etc, yeah, it's believable.

Re: Google Is 2B Lines of Code, All in One Place

#267

Post by thrownaway2424 » Thu, Sep 17, 2015, 3:51 AM UTC

jakub_g wrote: post #150

Some questions that immediately come to my mind: - What is the disk size of a shallow clone of a repo (without history)? - Can each developer actually clone the whole thing, or you do partial checkout? - Does the VCS support a checkout of a subfolder (AFAIK mercurial, same as git, does not support it)? - How long does it take to clone the repo / update the repo in the morning? Since people are talking about huge acro…

Most of your questions don't apply to the system described in this article. You do not clone the repository, you merely chdir into a vfs that is backed by a consistent view of the repository at a point in time, which view is served from a large distributed service that lives in Google datacenters alongside other Google services like Search, Maps, and Gmail. Because it is enormous and nobody clones it, it is also true that nobody partially clones it. You do not "checkout a subfolder" either.

Your last point is the only one that applies. If you want your view to advance from revision 123 to revision 125 it takes about a second to do so. If you have pending (not yet submitted) changes in your client, they might have to be merged with other changes, which can take a bit longer. If you have a really huge pending change, and your client is way behind HEAD, it might take a few tens of seconds to merge everything.

Re: Google Is 2B Lines of Code, All in One Place

#268

Post by bruckie » Thu, Sep 17, 2015, 3:52 AM UTC

jakub_g wrote: post #150

Some questions that immediately come to my mind: - What is the disk size of a shallow clone of a repo (without history)? - Can each developer actually clone the whole thing, or you do partial checkout? - Does the VCS support a checkout of a subfolder (AFAIK mercurial, same as git, does not support it)? - How long does it take to clone the repo / update the repo in the morning? Since people are talking about huge acro…

Most of these questions are answered in the talk. The tl;dr is that you don't clone or check out anything at all: instead, you use CitC to create a workspace, and the entire repository is magically available to you to view or edit.

This model precludes offline work, of course. But that's not much of a problem in practice.

Re: Google Is 2B Lines of Code, All in One Place

#269

Post by bruckie » Thu, Sep 17, 2015, 3:54 AM UTC

Locke1689 wrote: post #225

What I'd like to know and no one seems to mention: What's the experience like for teams not running a Google service and instead interacting with external users and contributors, e.g. the Go compiler or Chrome.

Many larger external projects are hosted in other repositories (Chrome and Android are well-known examples).

Smaller stuff (like, say, tcmalloc or protocol buffers) is usually hosted in Piper and then mirrored (sometimes bidirectionally) to an external repository (usually GitHub these days).

Re: Google Is 2B Lines of Code, All in One Place

#270

Post by sowbug » Thu, Sep 17, 2015, 3:55 AM UTC

Touche wrote: post #101

devit wrote: post #22

Earlier quoted context omitted.

The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted. The Linux kernel generally uses this policy for internal APIs for example.

> The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted. Having people unaware of a project's purpose making changes to its code sounds like a nightmare to me.

When that's the cultural norm, people adjust accordingly. Tools include liberal use of assertions, defensive tests, and most important, code reviews that catch and remove quirkiness.

It's a nice environment to work in. In addition to hastening Noogler onboarding, it also increases employee retention. If you are an expert in your project's codebase but get burned out, you can easily transfer to another project and be almost immediately productive.

Obviously, there's domain-specific knowledge that doesn't transfer easily or quickly from project to project. But that's quite different from self-inflicted code fragility; one's an asset and the other's a liability.