Live data from Hacker News

Debian Git Monorepo

blog.liw.fi

131–140 of 157 posts

Re: Debian Git Monorepo

#131
post #97

This needs a "April Fools" in the Headline so people don't waste time or energy reading it.

Definitely unfair to post this on the second of April without a disclaimer.

I think ideally we'd have month/year timestamps for every link submitted, but that's probably not going to happen, so you end up with instances like this, or months-old news being taken as new, etc.

I think the better solution, which I'm trying to do myself, is noting the date on the article before reading

Re: Debian Git Monorepo

#132
post #62

Earlier quoted context omitted.

Why use submodules when you can properly vendor the upstream git, and export/import commits without breaking hashes on either side? https://github.com/josh-project/josh We've been using josh at TVL for years and it's just amazing.

How does that work with upstreams that often needs patches?

It depends on exactly what workflow you're trying to accomplish!)

The parent comment seemed to be about a workflow where the repos are in sync. If you're carrying patches around and you don't want to export them, you are probably going to merging them. In either scenario if you have some patches you want to export and some you don't want to export, you'll either have to carry two trees, or do cherry-picking or something similar.

Re: Debian Git Monorepo

#133
post #2

I was always surprised that Perforce didn't have anything for open source, Git is basically eating their lunch because monorepo's suck so much with git; so people do things in a more fragmented and git-friendly way. Not that I really want to see a proprietary product succeed, but it's somewhat surprising that: A) We don't have anything better B) Perforce isn't trying to gain adoption by giving its software for free t…

For better or for worse, Perforce seems to see itself mainly in the niche of having binary or media files in source control. They are busy promoting themselves in game dev, automotive, etc, where this is a common pain point. Maybe they should do more to highlight their strengths with regards to monorepos, but apparently their marketing hasn't identified that as a strength, or thinks it isn't a worthwhile market.

Games I understand, but what makes automotive a place that uses a lot of binary files?

Re: Debian Git Monorepo

#134

April fool aside, what I really want is a Debian git monorepo of submodules, where each submodule points to the upstream git repository.

Please never use submodules for anything, they cause a litany of problems now and down the road (have you ever tried deleting or renaming one?) The best thing to do is to pretend they don't exist.

Re: Debian Git Monorepo

#135

I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB? To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third p…

Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.

it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)

Re: Debian Git Monorepo

#136

Earlier quoted context omitted.

Doesn't google have a lot of internal tooling to make that monorepo actually work for them? The issue isn't so much that large monorepos don't work, it's that you are trading one set of issues for another set of issues. And most popular open source tooling (and I assume all debian-specific tooling) is built around solving the issues of many repositories rather than solving the issues of monorepos.

> Doesn't google have a lot of internal tooling to make that monorepo actually work for them? probably tests coverage and workflow matters: you can work with monorepo if you can check how your change will impact rest of the ecosystem.

[deleted]

Re: Debian Git Monorepo

#137

April fool aside, what I really want is a Debian git monorepo of submodules, where each submodule points to the upstream git repository.

subtree might be better for exactly that? I have a project with about 100 repos, developing with them individually is so much saner, but every so often you want the one actual benefit of a monorepo: fast really-global search. So keeping a subtree-of-everything (using the bitbucket api to get the list of repos) and then just doing git-grep there when I want to see any use of something.

(Somewhere there's an apt hook for "grab sources for everything that apt installs"; having that extract the source and git commit it, or just get the upstream repo from the metadata and `git subtree add` (or pull) wouldn't take much...)

Re: Debian Git Monorepo

#138
Curious, what challenges would git still have to deal with to work with big repos?

500 GB isn't all that much data in 2024, as long as you can resume a broken network connection on fetch (like using HTTP range requests if using http, and when using git LFS), and while 15 million files is a pain in most file systems because of all the overhead, a relatively modern NVMe SSD should be able to cope with it just fine, but I don't know enough about the innermost internals of git on how it manages files. Is there something that's inherently slow/expensive that doesn't scale well with the number of files?

(Basically, I'm trying to find a non-rhetorical answer to the question of "Why not?")

Re: Debian Git Monorepo

#139

Putting aside that this is an April Fool's joke, I like the last part: > This time, I’m cruel to Git: can it handle a repository of this size? In 2009 it could not. In 2024 it can. That really hits home for me. Really: Think about this repo for a moment. 500 GB. 15 miiiiiiiiiiiiiiilion files. It is crazy to think that a "vanilla" git repo can handle it. Applause for the Git team!

I find the Gentoo ebuild repository, mirrored at http://github.com/gentoo/gentoo often is a stress test for git clients due to the sheer number of commits.

It's interesting to see the definition of scale stretched in all kinds of different directions.

Post reply on HN