Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

71–80 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#71

Post by precommunicator »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Context: happy monorepo user.

1, 2 and 3: Use separate dependencies for each package, so this doesn't happen. Use e.g. GitHub Actions or another CI/CD file filtering wisely: if a file is needed by two packages, tests for both packages needs to run whenever it's changed, before merging, in addition to usual end-to-end tests. Have vulnerable dependencies alerting and make sure to upgrade it everywhere it occurs.

2: Also have some guidelines on that and enforce it either automatically or manually in PRs.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#72

Post by wdb »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Personally, I am looking forward to switch to a monorepo as it makes things a lot easier. Makes testing a lot easier when you don’t need to deal with 70 repositories to test something. Also it’s easier to ensure dependencies such as API libraries are up to date in each service. Quicker feedback whether code changes break the things. Now I have to wait at least 24 hours to find if my PR that I merged breaks things.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#73

Post by [deleted] »

gresrun wrote: post #44
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

[deleted]

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#74

Post by graveltongue »

jmillikin wrote: post #46
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

https://opensource.google/documentation/reference

The third party documentation is public, one-version policies exist but they are exemptions.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#75

Post by password11 »

gresrun wrote: post #44
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

>> 3. It encourages a ton of code churn with very low signal.

> 3) “Code churn” is a very uncharitable description of day-to-day maintenance of an active codebase.

Also implicit in the discussion is the fact that Google and other big tech companies performance review based on "impact" rather than arbitrary metrics like "number of PRs/LOCs per month". This provides a check on spending too much engineer time on maintenance PRs, since they have no (or very little) impact on your performance rating.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#76

Post by phphphphp »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

You’re describing bad habits as if they’re a forgone conclusion. Repository-level separation between code makes certain bad habits impossible so a sloppy team will be more effective with many-repos because they physically can’t perform an entire class of fuck-ups but there’s lots of organisations where these fuck-ups… just don’t happen, and so the co-locating code in a monorepo isn’t a concern.

If your organisation can’t work effectively within a monorepo then you should absolutely address the problem, either by fixing the problematic behaviour or by switching away from a monorepo. The problem isn’t monorepos, the problem is monorepos in your organisation.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#77

Post by admax88qqq »

spankalee wrote: post #63
lallysingh wrote: post #55

Earlier quoted context omitted.

No you use automated systems to do the change. https://mobile.twitter.com/obeattie/status/10804969557537505...

Even so, the cost is often outrageously high. No to mention that if you're the first team to import a third_party library, you own it and other teams can add arbitrary cost to you updating it. You have to be very aggressive with visibility and SLAs to work around this.

You're basically just describing all the pain with pulling in a dependency regardless of monorepo or not.

If the third party dependency does not add enough value to justify the cost then don't add it.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#78

Post by bananapub »

lopkeny12ko wrote: post #59
jmillikin wrote: post #46

Earlier quoted context omitted.

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

> There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd hav…

> Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd have to add a new dependency @myjslib_v3_1_0 used only by Bar. After two years you'd have 10 unique dependencies for 10 versions of myjslib in the monorepo.

you're imagining a situation and speculating up a problem that might occur in that imaginary situation. in reality, no one does the thing you said - you don't add random deps on random external javascript libraries that don't have sane versioning stories.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#79

Post by gajjanag »

password11 wrote: post #75
gresrun wrote: post #44

Earlier quoted context omitted.

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

>> 3. It encourages a ton of code churn with very low signal. > 3) “Code churn” is a very uncharitable description of day-to-day maintenance of an active codebase. Also implicit in the discussion is the fact that Google and other big tech companies performance review based on "impact" rather than arbitrary metrics like "number of PRs/LOCs per month". This provides a check on spending too much engineer time on mainten…

> based on "impact" rather than arbitrary metrics

Umm, from whatever I have seen in big tech "impact" is also fairly arbitrary. It all is based on how cozy one is with one's manager, skip manager, and so on. More accurate is "perception of impact".

Especially as it gets more and more nebulous at higher levels.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#80

Post by bananapub »

Karellen wrote: post #56

> The Google codebase includes approximately one billion files and has a history of approximately 35 million commits spanning Google’s entire 18-year existence. Wait, that's an average of nearly 30 new files per commit. Not 30 files changed per commit, but whatever changes are happening to existing files, plus 30 brand new files. For every single commit. Although... > The total number of files also includes source fi…

you're misunderstanding a bunch of things.

> The total number of files also includes source files copied into release branches

I guess you haven't used Perforce or similar. a branch is a sparse copy of just the changed files/directories. they are not used very much.

> files that are deleted at the latest revision

so it means "one billion files have existed in the history repo, some are currently deleted".

> I don't think it's entirely clear what the paper even means when it talk about "a file" in a source code repository,

seems pretty clear - a source code repo has lots of files. at the most recent revision, some exist, some were deleted in some past revision. more will be added (and deleted) in later revisions.

it's very much not the same model as git.

hope that clears things up.

Post reply on HN