Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a new generation of source control

rookout.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: We need a new generation of source control

#21

Post by headcanon »

kadendogthing wrote: post #4

As I've stated in another post on here, what's the point of these articles? It just says everything sucks, but doesn't really dive into why or how we could possibly fix any issues they may directly point out. Also it kind of sounds like the author really doesn't have any idea what GitLab is or does, so maybe he should check it out. But allow me to retort these bald assertions presented in the article: Monorepos are g…

One thing about GitLab tooling is that they have features that apply only on a per-repo basis, for example GitLab CI.

Suppose for example we have 2 distinct projects - a backend and frontend, which each have their own testing and deployment strategy. GitLab CI only allows one CI pipeline config per-repository. While we could take care of that with scripting, that can easily get out of hand as we increas the number of distinct "projects", if we wanted to maintain a monorepo. So the tooling encourages us to have separate repos.

However if we do that, since we don't have that convenient single commit hash that a monorepo gives us, then we don't have a good way to ensure that the deployments between projects are synced up, and rollbacks are far more complicated.

Its a contrived example (for instance we could switch to a different CI system and mitigate this issue), but it seems to me that whatever an organization chooses, mono- or poly-repo, they have to build complicated custom configurations and tooling to get over whatever tradeoffs their decision has. And as the number of logical projects (repos, submodules, etc.) and the commit rate increases, then the tooling has to increase in complexity to handle issues of scale.

So I guess the open question is, is there a way we can somehow have both without spending a bunch of engineering cycles writing custom configs and tools?

Re: We need a new generation of source control

#22

Post by arianvanp »

The argument of why monorepos suck seems to largely rely on "CI Sucks" in this article. But I beg to differ. Monorepos only work in combination with a build system that tracks dependencies carefully.

I contribute a lot to Nixpkgs, which is a monorepo with almost 50000 subcomponents [1], but because the build tool and CI track changes through hashes, changing a package only triggers rebuilds of other packages that depend on it and builds are super quick. It accomplishes this by heavily caching previous builds and sharing those between all builders.

No, monorepos are not going to work with a CI and build tool that always builds everything from scratch and does no caching. Instead, you should pick the right tool for the job, and go with a build system like Nix, Buck, Bazel or Please which were designed with monorepos in mind.

I think the second point the author makes, but only very briefly, is way more important to look at. Is git itself up to the job for such large repositories? One problem I've started running into in nixpkgs is that `git blame` takes considerable time to even execute, due to the enormous volume of commits in the repository. I would love to see a version control system that is optimised for storing lots of loosely connected components, and has better support for partial checkouts. I haven't found it yet, and I would love to hear what others are using for this.

I hear facebook has some modification of mercucurial. And Google probably created something themselves in-house. But is there anything open-source that supports these workflows at scale?

[1] https://repology.org/repository/nix_unstable

Re: We need a new generation of source control

#24

Post by masklinn »

weberc2 wrote: post #8
mikece wrote: post #5

I like the idea of creating a source control protocol that can be implemented with any number of tools rather than having wars over particular implementations of source control products. (And would Git really have beaten Mercurial if GitHub had been HgHub instead? GitHub's success was more about process than the technology of Git, IMO.)

> And would Git really have beaten Mercurial if GitHub had been HgHub instead? GitHub's success was more about process than the technology of Git, IMO. Hg is a much better user experience than git, that's for sure. Git won because of Github, which may have beaten any HgHub simply because Git has an actual API while Mercurial's "API" is "use subprocessing". In other words, if Mercurial gave a damn about the developer…

> which may have beaten any HgHub simply because Git has an actual API

Git doesn't though. A bunch of shell scripts calling shell scripts calling a few native binaries is pretty much "use subprocessing". libgit came much later, it wasn't part of the original git.

However what git did provide was an open, stable, fairly simple and officially supported physical model with which you could easily interact directly, and protocols which either worked on that (file and "dumb http") or a relatively simple exchange protocol (the "pack protocol" https://github.com/git/git/blob/9b011b2fe5379f76c53f20b964d7...).

Hell, if anything hg's always provided more API than git, the extension model wouldn't be possible without it e.g. stdout coloration could be an hg plugin while it had to be implemented in each git command.

Re: We need a new generation of source control

#25

Post by rabi_penguin »

I don't know if I really follow the conclusion from this blog post although I sympathize with the complaints. Let's take one point: " In fact, not only will Git CI tools rebuild and redeploy your entire repo, they are often built explicitly for multi-repo projects." This seems patently wrong. On buildkite, which we use, you can explicitly set up build steps to trigger based on directory patterns.

In my experience on teams at growing companies, I've seen pain points around continuous integration, configuration management, integration testing, dev/prod parity, feature flagging and releasing, provisioning staging servers in terms of pure tech/infra issues. Beyond that, I've seen more pressing general organizational issues around tech debt, software design collaboration, architectural debt, code review processes -- these are all pressing and valid concerns. But I just find the conclusions of this blog post flat out wrong. To conflate an unsatisfactory CI choice and configuration (which is totally reasonable) with a failure of version control is a pretty serious one. It doesn't fully disprove the thesis, but it certainly doesn't lend it support.

If you've installed a wheel onto a poorly set up suspension and get handling issues, does it mean you should reinvent that wheel, or does it mean you should check if your suspension may need some tuning?

Re: We need a new generation of source control

#26

Post by brightball »

hliyan wrote: post #19

Are we mistaking a dependency control problem as a revision control problem? In a previous life, before microservices, CI/CD etc. existed, we did just fine with 20-30 CVS repositories, each representing a separate component (a running process) in a very large distributed system. The only difference was that we did not have to marshal a large number of 3rd party dependencies that were constantly undergoing version cha…

You’re not wrong. Part of it is the willingness of people to reach for a dependency that amounts to a few lines of code to avoid.

It would be nice if there was a tool that could help you identify just how much of each dependency you actually depend on so you could trim it.

Re: We need a new generation of source control

#27

Post by tempguy9999 »

hashkb wrote: post #10
klodolph wrote: post #7

Earlier quoted context omitted.

> Git is the best source control system ever. To be clear, I'm not disagreeing. But it is simply not good enough. Any new generation of source control needs to be able to do things that are difficult with Git, and Git simply isn't extensible enough. Microsoft has a Git VFS, and there's Git LFS, but this just doesn't go far enough. There are good technical reasons why you would use Perforce or even Subversion these da…

> There are good technical reasons why you would use Perforce or even Subversion days. Can you say more? What are some of those reasons? Or link to some data or examples?

Simplicity. I understood SVN immediately but I'm still struggling with Git.

It's only one thing and perhaps the only one, but it's a huge one. IMO anyway.

Re: We need a new generation of source control

#28

Post by hliyan »

brightball wrote: post #26
hliyan wrote: post #19

Are we mistaking a dependency control problem as a revision control problem? In a previous life, before microservices, CI/CD etc. existed, we did just fine with 20-30 CVS repositories, each representing a separate component (a running process) in a very large distributed system. The only difference was that we did not have to marshal a large number of 3rd party dependencies that were constantly undergoing version cha…

You’re not wrong. Part of it is the willingness of people to reach for a dependency that amounts to a few lines of code to avoid. It would be nice if there was a tool that could help you identify just how much of each dependency you actually depend on so you could trim it.

Is there something that is akin to development-time tree-shaking (as opposed to build time)? i.e. you pull a copy of the specific library functions directly into your source?

Re: We need a new generation of source control

#29

Post by organsnyder »

hliyan wrote: post #19

Are we mistaking a dependency control problem as a revision control problem? In a previous life, before microservices, CI/CD etc. existed, we did just fine with 20-30 CVS repositories, each representing a separate component (a running process) in a very large distributed system. The only difference was that we did not have to marshal a large number of 3rd party dependencies that were constantly undergoing version cha…

My preferred approach for a microservice architecture:

- Contract-first API development

- All API contract definition files (OpenAPI/Swagger, .proto, .wsdl...) in a single repo, which has a CICD pipeline to bundle them into artifacts for various platforms (Maven, Nuget, NPM, gem...)

- Consumers and producers import the "api-contracts" dependency; this is the only coupling between components

- Consumers and producers both generate necessary code (server stubs, client libraries) at build time

IMHO, if your service clients have dependencies on implementations of APIs rather than just the definitions, you're not realizing the key benefit of microservices (or SOA).

Re: We need a new generation of source control

#30

Post by joshuamorton »

brightball wrote: post #26
hliyan wrote: post #19

Are we mistaking a dependency control problem as a revision control problem? In a previous life, before microservices, CI/CD etc. existed, we did just fine with 20-30 CVS repositories, each representing a separate component (a running process) in a very large distributed system. The only difference was that we did not have to marshal a large number of 3rd party dependencies that were constantly undergoing version cha…

You’re not wrong. Part of it is the willingness of people to reach for a dependency that amounts to a few lines of code to avoid. It would be nice if there was a tool that could help you identify just how much of each dependency you actually depend on so you could trim it.

These things all exist if you use something like bazel/pants/buck to manage your dependencies. When you can construct a DAG of the entire dependency structure you can see exactly how much you depend on any given thing (and get fun dot-graphs of it!). But that requires being precise with dependency declaration in a way that a lot of people don't want to be.
Post reply on HN