Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debian Git Monorepo

blog.liw.fi

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Debian Git Monorepo

#141

Post by jorvi »

zer00eyz wrote: post #114
jorvi wrote: post #52

Earlier quoted context omitted.

Github / Gitlab would work great and reduce friction for kernel contribution. The refusal to switch or provide the alternate path is mostly inertia.

I think you have a story eyed view of github... You might want to go read Linus own thoughts on the matter in the linux GitHub backup repo Git is a tool built for a project of the kernels scope, scale and organization. Github is a thin web interface over top of that, it cuts some corners here and there and gets opinionated about how you should manage code (pull requests). Think of it this way: most git hub projects e…

Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. People see a bit of code they want to improve, they improve it, and then instead of just opening a PR (and perhaps iterating on it), now they have to learn a weird ancient e-mail workflow and will probably get chewed out for messing it up.

No one is saying to take away mail-in patches but it is positively archaic.

Re: Debian Git Monorepo

#142

Post by francocalvo »

Xeamek wrote: post #133
wongarsu wrote: post #87

Earlier quoted context omitted.

For better or for worse, Perforce seems to see itself mainly in the niche of having binary or media files in source control. They are busy promoting themselves in game dev, automotive, etc, where this is a common pain point. Maybe they should do more to highlight their strengths with regards to monorepos, but apparently their marketing hasn't identified that as a strength, or thinks it isn't a worthwhile market.

Games I understand, but what makes automotive a place that uses a lot of binary files?

Probably tons of 3D models

Re: Debian Git Monorepo

#143

Post by bluGill »

JackSlateur wrote: post #109
bluGill wrote: post #75

Earlier quoted context omitted.

> I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. In abstract I agree. However when I'm trying to get my code working having test failures in code that isn't even related to the problem I'm working on is annoying and I can't switch tasks to work on this new failure when the…

How could broken code (or broken tests) be merged in master ? That is a rhetorical question, of course it happends, and of course this is the root issue you would be facing

As you may be aware, 100% broken code cannot be merged. However code that works 99.99% of the time can be merged and then weeks later it fails once but you rebuild and it passes. There are a lot of different ways this can happen.

Re: Debian Git Monorepo

#144

Post by layer8 »

radicality wrote: post #121
gtirloni wrote: post #47

Earlier quoted context omitted.

> It’s a nightmare to handle and use Quite the opposite, IMHO. Juggling dozens of PRs across many repos for a simple change is the definition of a nightmare.

Agreed. For the last three years I’ve been working at a company with literal thousands of repos, and it’s an actual nightmare compared to a well maintained monorepo (eg Facebook/Meta). Sometimes updating some library code means I do 50+ PRs across random repos.

You’re comparing two extremes, and your last sentence just indicates that the modules don’t seem to be properly divided along stable abstraction boundaries.

Re: Debian Git Monorepo

#145

Post by thaliaarchi »

sitzkrieg wrote: post #130
throwaway2037 wrote: post #110

Putting aside that this is an April Fool's joke, I like the last part: > This time, I’m cruel to Git: can it handle a repository of this size? In 2009 it could not. In 2024 it can. That really hits home for me. Really: Think about this repo for a moment. 500 GB. 15 miiiiiiiiiiiiiiilion files. It is crazy to think that a "vanilla" git repo can handle it. Applause for the Git team!

i used to work in git analytics and saw the most unimaginable repos. 500gb repos with one commit. repos where the source was in the commit message. repos with millions of commits consisting of bogus utf8, and so on. repos where they dont use branches, only remotes, etc etc. every usecase annihilated some old startup mvp code somewhere it felt like we wrote our own in memory client but even before that, stock git clie…

Do you still have a record of any of those weird case repos? Perhaps reported as issues somewhere? I’m working on a tool like git filter-repo and Reposurgeon combined and would love stress tests and edge cases like these.

Re: Debian Git Monorepo

#146

Post by tazjin »

mananaysiempre wrote: post #103
Zambyte wrote: post #82

Earlier quoted context omitted.

> it is close to how nixpkgs works Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition. The Nix equivalent to this joke would be storing all of the pac…

I’d say it is close. A purer example would be gittup[1], which has its (very few) upstreams as submodules. But ultimately, Git submodules are just URLs and commit hashes stored in .gitmodules in the repository root, which doesn’t seem all that different from the URLs and hashes of contents passed to fetchgit et al. in Nixpkgs. The capability of being able to patch an arbitrary package locally and have every transitiv…

Things don't "fall out of" the binary cache. Garbage collecting it is an active research area.

Re: Debian Git Monorepo

#147

Post by erik_seaberg »

gtirloni wrote: post #88
sethammons wrote: post #74

Earlier quoted context omitted.

that tells me you prematurely split something. I worked at a high scale, high available system with multiple dozens of teams, each with several repos. If we had to sync changes across repos, we joined the repos because they were, well, joined. In the early days, we had shared libraries and keeping those in sync was a pita. Then we realized we should not be doing what we were doing. The flow should be: (1) update a li…

> (2) code that wants/needs the new functionality updates the usage at its leisure. That's how you end up with infinite support costs in a large org. I think the point is that these changes are done atomically.

You can't deploy a distributed system atomically, and the closer you get the more risk you're taking with prod. The old and new versions need to coexist for at least a couple of weeks in case of rollbacks.

Re: Debian Git Monorepo

#148

Post by m_niedoba »

MenhirMike wrote: post #138

Curious, what challenges would git still have to deal with to work with big repos? 500 GB isn't all that much data in 2024, as long as you can resume a broken network connection on fetch (like using HTTP range requests if using http, and when using git LFS), and while 15 million files is a pain in most file systems because of all the overhead, a relatively modern NVMe SSD should be able to cope with it just fine, but…

Git does pretty well on large repos, when you use sparse checkout and Git LFS. It makes Git behave like a centralized VC system. https://www.anchorpoint.app/blog/scaling-git-to-1tb-of-files...

Re: Debian Git Monorepo

#149

Post by zer00eyz »

jorvi wrote: post #141
zer00eyz wrote: post #114

Earlier quoted context omitted.

I think you have a story eyed view of github... You might want to go read Linus own thoughts on the matter in the linux GitHub backup repo Git is a tool built for a project of the kernels scope, scale and organization. Github is a thin web interface over top of that, it cuts some corners here and there and gets opinionated about how you should manage code (pull requests). Think of it this way: most git hub projects e…

Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. People see a bit of code they want to improve, they improve it, and then instead of just opening a PR (and perhaps iterating on it), now they have to learn a weird ancient e-mail workflow and will probably get chewed out for messing it up. No one is saying to take away mail-in patches but it is positively arc…

>> Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution.

Is it friction? Or is it a filter?

You might remember being a kid and there was the sign in front of the ride that said "you must be at least this high to ride".... The kernel dev process isnt for casuals. It's designed that way.

There's a lot of folks out there who have popular projects on GitHub who are over the endless stream of BS from AI generated pull requests.

You should really dig in deep to what goes on with the kernel, the work flow, why it is that way and why GitHub is outright incapable of supporting kernel dev (there are reasons).... Your going to look at git in a very different way and many of githubs features are gonna feel on par with linkedin adding twitch style videos and zoom adding mail features...

Re: Debian Git Monorepo

#150

Post by BlackFly »

ghosty141 wrote: post #80
BlackFly wrote: post #31

Earlier quoted context omitted.

The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on tha…

> Similarly, once you are too popular a library in a monorepo, you will never do any atomic breaking API changes since it would require updating too many downstreams. This happens no matter which repo type. Even worse if a project chooses to update after a while, its far more painful having to do the changes after losing the context you had when you did the original changes. If you want a monorepo, libraries being on…

This doesn't happen in a poly repo because you can just do it. You release version 2.0.0 of something and downstreams update at their own pace. Diverging as you call it.

But this isn't a problem. If 1.0.0 is a finished product then why do you ever need to move to 2.0.0 if you don't need the new features?

The issue in the monorepo is that if you are too popular the change must happen all at once or with copying (fake versioning, like people who version excel files by suffixing with dates), which places pressure on maintainers to not fix design mistakes.

It isn't a feature of the monorepo because you can still diverge by copying, forking or merely stopping support for the old library and this becomes more and more necessary at scale: you lose the feature you thought you wanted the monorepo for.

Post reply on HN