Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

111–120 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#111

Post by saagarjha »

forgotusername6 wrote: post #29
scarface74 wrote: post #25

Earlier quoted context omitted.

If you had every line of code that Google wrote, what would you do with it? But I found this discussion on HN. https://news.ycombinator.com/item?id=11790438

I imagine looking for vulnerable areas of the code might be something people would be interested in doing. Maybe start with login or billing or something. You could also look at recent activity to spot new, unannounced projects. You could use blame to find who wrote what and target them for anything from job offers to social engineering attacks.

People can find vulnerabilities without the code too.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#112

Post by joshuamorton »

jmillikin wrote: post #46
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

There's always been a very strong one version policy, multiple versions are usually only allowed to coexist for weeks or months, and are usually visibility restricted.

This prevents situations where "Gmail" ends up bundling 4 different, mildly incompatible versions of MySQL or whatever, and the aggravation that would cause. Or worse, in c++ you get ODR violations due to a function being used from two versions of the same library.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#113

Post by doctor_eval »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

I agree with every single point you made. Unfortunately, it's one of those discussions that is never going to be resolved because like so much else, it's difficult to find common ground when there are competing priorities.

My point is that in reality, we use what best matches our knowledge, experience and perception and prioritisation of the problems. I, for one, believe that a monorepo is dangerous for small teams because it encourages coupling - not only do I believe it, but I saw it with my own eyes. It also creates unnecessary dependency chains. Monorepos contribute to a fallacy that every dependent on an object must be immediately updated or tech debt happens. But that's not even remotely given.

In any case, companies like Google and Amazon have more than enough resources to deal systematically with the problems of a monorepo. I'm sure they have entire teams whose job it is to fix problems in the VCS. But for small teams I remain unconvinced that it is a good idea. We shouldn't even be trying to do the things the big guys do, unless we want to spend all our time working on the tools instead of our businesses.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#114

Post by barbazoo »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression

Is this convention for monorepos to all share the same dependencies? Does monorepo imply monolith? Surely one could have dependencies per "service" for example a python app with its own pipfile per directory.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#115

Post by taeric »

jmillikin wrote: post #46
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

I think the catch, is that it isn't just third-party dependencies that are of concern. In particular, at a certain size, you are best off treating every project in the company as a third party item. But, that is typically not what you are wanting with source dependencies.

You can see this some with how obnoxious Guava was, back in the day. It seems a sane strategy where you can deprecate things quickly by getting all callers to migrate. This is fantastic for the cases where it works. But, it is mind numbingly frustrating in the cases where it doesn't. Worse, it is the kind of work that burns out employees and causes them to not care about the product you are trying to make. "What did you do last month?" "I managed to roll out an upgrade that had no bearing on what we do."

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#116

Post by taeric »

jcranmer wrote: post #88
lopkeny12ko wrote: post #59

Earlier quoted context omitted.

> There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd hav…

So why is this a problem for a monorepo but not the multi-repo? It seems to me that the major difference is that in a multi-repo, you'd be more likely to be oblivious to the multitude of dependency issues than you are in the monorepo... and to be honest, that actually sounds like a bad thing to me, because it means you're sweeping legitimate issues underneath the rug.

In a multi-repo, you don't build source dependencies between projects.

You can do this with a mono, as well. However, the conceit is that "in the same repo" means you can "change them together." It is very very tempting that "went out as a single commit" means that it went out fine. Which, just isn't something you see in a multi world.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#117

Post by smrtinsert »

zelphirkalt wrote: post #91
jmillikin wrote: post #46

Earlier quoted context omitted.

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

I guess the question then becomes: Is it worth all the extra tooling required to manage a monorepo properly?

The answer of course is no, but since they have a stable search product that supplied unlimited money, they were able to stubbornly stick to that decision.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#118

Post by taeric »

myko wrote: post #62
lopkeny12ko wrote: post #59

Earlier quoted context omitted.

> There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd hav…

> Sure, but this is unsustainable. Not exactly unsustainable considering Google has been very successful with this approach!

I want to think they have. But... this is also why they kill older products. The cost of keeping the lights on is greatly elevated when keeping the lights on means keeping up with the latest codes.

This is absolutely no different from buildings. If you had to keep every building up to date with the latest building codes, you would tear them down way way way more often.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#119

Post by jcranmer »

taeric wrote: post #116
jcranmer wrote: post #88

Earlier quoted context omitted.

So why is this a problem for a monorepo but not the multi-repo? It seems to me that the major difference is that in a multi-repo, you'd be more likely to be oblivious to the multitude of dependency issues than you are in the monorepo... and to be honest, that actually sounds like a bad thing to me, because it means you're sweeping legitimate issues underneath the rug.

In a multi-repo, you don't build source dependencies between projects. You can do this with a mono, as well. However, the conceit is that "in the same repo" means you can "change them together." It is very very tempting that "went out as a single commit" means that it went out fine. Which, just isn't something you see in a multi world.

> In a multi-repo, you don't build source dependencies between projects.

In my experience with software projects, this is very much not the case. It's one of the main reasons I'm such a big fan of monorepos--I have been burned way too many times by the need to make atomic commits involving separate repositories.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#120

Post by taeric »

jcranmer wrote: post #119
taeric wrote: post #116

Earlier quoted context omitted.

In a multi-repo, you don't build source dependencies between projects. You can do this with a mono, as well. However, the conceit is that "in the same repo" means you can "change them together." It is very very tempting that "went out as a single commit" means that it went out fine. Which, just isn't something you see in a multi world.

> In a multi-repo, you don't build source dependencies between projects. In my experience with software projects, this is very much not the case. It's one of the main reasons I'm such a big fan of monorepos--I have been burned way too many times by the need to make atomic commits involving separate repositories.

If you have multiple repos, you can't have an atomic commit between them. Pretty much period. I'm scared to hear what you mean on that.

Ideally, all tooling makes the separate nature of the projects transparent. They should test separately. They should deploy separately. If that is not the case, then yes, they should be in the same repo.

Post reply on HN