Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

161–170 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#161

Post by redditor98654 »

yazaddaruvala wrote: post #154
safog wrote: post #125

Earlier quoted context omitted.

There are no presubmits that prevent breaking changes from "going into live". If some shared infra updates are released, the merge from live breaks for multiple individual teams rather than preventing the code from getting submitted in the first place.

I don’t agree with your assessment. “Merging to live” builds and tests all packages that depend on the update. So for example, building the new JDK to live will build and test all Java packages in previous live, all of them need to pass their package’s tests, only then will the JDK update be “committed into live”. The only difference is that Google runs all the presubmits / “dry run to live checks” in the CL workflow…

If it is a new major version of the JDK, then no. Because no existing code would have a dependency on that new JDK major version so nothing gets rebuilt against the new JDK build in live and it will be committed.

If it is a new commit of an existing major version, then yes, existing code in live will be rebuilt before the packaged is committed to live.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#162

Post by dx034 »

skybrian wrote: post #129
yongjik wrote: post #84

Earlier quoted context omitted.

Effort to update something is high because there's a lot of code, not because it's in a monorepo. Updating the same code scattered across multiple repositories takes as much work in the best case. More realistically, some copy of the same code will stay unupdated because the cost to track down every repository in the company is too much.

When you say it's "as much work" there's an assumption the code is still used. This was years ago, but when I was doing migrations at Google we sometimes had to deal with abandoned or understaffed and barely maintained code. (Sometimes by deleting it, but it can be unclear whether code by some other team is still useful.) If you're not responsible for fixing downstream dependencies then you don't need to spend any ti…

Sounds great to me because you are forced to delete code that's not in use anymore. Without the monorepo, that code would still be there with old libraries that are potentially insecure.

Deleting code that is not being used anymore happens way too rarely in my opinion.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#163

Post by dx034 »

ASinclair wrote: post #93
jmillikin wrote: post #46

Earlier quoted context omitted.

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

There’s a policy against multiple versions of third party dependencies. Though there is a mechanism for exceptions.

Sounds sane, usually you don't want multiple versions just because people were too lazy to keep code up to date. But in some instances it's probably worth supporting several major versions across the whole org.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#164

Post by RamblingCTO »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

1 and 2 could be solved by using proper gradle multi-module projects and tests. So I would say this is a problem of tooling of the language you're using. This is one of the reasons why I still can't understand how people operate with inferior ecosystems like node in the backend and I also wish go would have these things.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#165

Post by dx034 »

throwaway2037 wrote: post #98
lopkeny12ko wrote: post #57

Earlier quoted context omitted.

You should ask your colleagues who work in critical industries like banking and healthcare how much of their software stack depends on things that haven't been patched in more than 20 years ;)

"critical industries like banking and healthcare". What a red herring. This comment reads like ChatGPT was trained on Reddit forums. 99% of the software in those industries runs "inside the moat" where security doesn't matter. I am still running log4j from 10 years ago in lots of my stack, and it is the swiss cheese of software security! Who cares! It works! I'm inside the moat! If people want to do dumb black hat st…

I don't think it matters much if you're inside the moat. Running vulnerable software inside the moat makes it very easy for an attacker to move laterally once they're in. Patching everything where possible reduces the blast radius of an attack massively.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#166

Post by aimxhaisse »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

I find 1) to be a good property assuming you have some safeguards or rollback procedure, at a cultural/code ownership level it moves the efforts of shared-code changes on the person doing them rather than on the ones depending on shared code, which reduces communications, frustration points and increase responsibility.

For instance in multi-repo environments I've often seen this pattern: own some code, bump an internal dependency to a new version, see it break, ask the person maintaining it what's us, realize this case wasn't taken into account, few back and forth before finding an agreement.

On the other hand in mono-repo environments, it's usually more difficult to introduce a wide changes as you face all consequences immediately, but difficulty is mainly a technical/engineering difficulty rather than a social one, and the outcome is better than the series of compromises made left and right after a big multi-repo change.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#167

Post by xorcist »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

That sounds like good arguments for monorepos. Bumping a js package that is used in several places should break the build, that how you test it. It sounds like the fallout of the version bump was caught already on the next build, so hopefully it didn't make it into the master-equivalent branch.

Compare that with hundreds of tiny repos, each with their own little dependency system. Testing a version bump across the board before mainlining it is much more involved and you are more likely to hit stuff in production which should have been caught in test.

The other two points sounds more like cultural issues which may touch on branch strategies, code review, and what's expected of a developer. Those mostly cultural issues that overlaps with technical are hard in a way that repository strategy isn't.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#168

Post by spion »

yongjik wrote: post #158
spion wrote: post #139

Earlier quoted context omitted.

But with a multi-repo, its possible to e.g. upgrade the dependency just for a single service that has an immediate need for the upgrade, isn't it?

The flip side is that services with an immediate need will get upgraded, and others won't, and six months later you will be saying "Why am I still seeing this bug in production, I already fixed it three times!" Of course, the problem can be mitigated by a disciplined team that understands the importance of everybody being on the same page on which version of each library one should use. On the other hand, such a team…

I don't think that's quite true. In my experience multi-repos have the edge here.

If you have one key dependency update with a feature you need, but you need substantial code updates and 80 services depend on it, that may be impossible to pull off no matter what. Comparatively, upgrading one by one may not be easy, but at least its possible.

The importance of everyone being on the same page with dependencies might just be a limitation of monorepos rather than a generally good thing. Some services might just not need the upgrade right now. Others may be getting deprecated soon, etc.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#169

Post by zelphirkalt »

safog wrote: post #125
yazaddaruvala wrote: post #22

Having worked at Google and Amazon. Honestly their systems are almost identical. Amazon just creates a monotonically increasing watermark outside the “repo”. Google uses “the repo” to create the monotonically increasing watermark. Otherwise, Google calls it “merge into g3” Amazon calls it “merge into live”. Amazon has the extra vocabulary of VersionSets/Packages/Build files. Google has all the same concepts, but just…

There are no presubmits that prevent breaking changes from "going into live". If some shared infra updates are released, the merge from live breaks for multiple individual teams rather than preventing the code from getting submitted in the first place.

With an appropriately configured CI pipeline, submitted / pushed code does not go live anyway, unless all tests and other checks pass. Unless a test case is missing, which can happen in a mono repo just as well, the code is always checked for the defect.
Post reply on HN