Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We shrunk our Javascript monorepo git size

jonathancreamer.com

Post reply on HN
Print view View original

191–200 of 226 posts

Re: We shrunk our Javascript monorepo git size

#191

Post by hinkley »

lopkeny12ko wrote: post #173
develatio wrote: post #74

Hacking Git sounds fun, but isn't there a way to just not have 2.500 packages in a monorepo?

This was exactly my first thought as well. This seems like an entirely self-manufactured problem.

When you have hundreds of developers you’re going to get millions of lines of code. Thats partly Parkinson’s Law but also we have not fully perfected the three way merge, encouraging devs spread out more than intrinsically necessary in order to avoid tripping over each other.

If you really dig down into why we code the way we do, the “best practices” in software development, about half of them are heavily influenced by merge conflict, if not the primary cause.

If I group like functions together in a large file, then I (probably) won’t conflict with another person doing an unrelated ticket that touches the same file. But if we both add new functions at the bottom of the file, we’ll conflict. As long as one of us does the right thing everything is fine.

Re: We shrunk our Javascript monorepo git size

#192

Post by hinkley »

Cthulhu_ wrote: post #108
develatio wrote: post #74

Hacking Git sounds fun, but isn't there a way to just not have 2.500 packages in a monorepo?

Yeah, have 2500 separate Git repos with all the associated overhead.

Changing 100 CI pipelines is a giant pain in the ass. The third time I split the work with two other people. The 4th time someone wrote a tool and switched to a config file in the repo. 2500 is nuts. How do you even track red builds?

Re: We shrunk our Javascript monorepo git size

#193

Post by js2 »

eviks wrote: post #7

upd: silly mistake - file name does not include its full path The explanation probably got lost among all the gifs, but the last 16 chars here are different: > was actually only checking the last 16 characters of a filename > For example, if you changed repo/packages/foo/CHANGELOG.md, when git was getting ready to do the push, it was generating a diff against repo/packages/bar/CHANGELOG.md!

> file name does not include its full path

No, it is the full path that's considered. Look at the commit message on the first commit in the `--full-name-hash` PR:

https://github.com/git-for-windows/git/pull/5157/commits/d5c...

Excerpt: "/CHANGELOG.json" is 15 characters, and is created by the beachball [1] tool. Only the final character of the parent directory can differntiate different versions of this file, but also only the two most-significant digits. If that character is a letter, then this is always a collision. Similar issues occur with the similar "/CHANGELOG.md" path, though there is more opportunity for differences in the parent directory.

The grouping algorithm puts less weight on each character the further it is from the right-side of the name:

  hash = (hash >> 2) + (c
Hash is 32-bits. Each 8-bit char (from the full path) in turn is added to the 8-most significant bits of hash, after shifting any previous hash bits to the right by two bits (which is why only the final 16 chars affect the final hash). Look at what happens in practice:

https://go.dev/play/p/JQpdUGXdQs7

Here I've translated it to Go and compared the final value of "aaa/CHANGELOG.md" to "zzz/CHANGELOG.md". Plug in various values for "aaa" and "zzz" and see how little they influence the final value.

Re: We shrunk our Javascript monorepo git size

#194

Post by tazjin »

avianlyric wrote: post #186
tazjin wrote: post #117

Earlier quoted context omitted.

> What's up with folks in Europe that they can't clone a big repo, but others can? They might be in a country with underdeveloped internet infrastructure, e.g. Germany))

I do t think there’s any country in Europe with internet infrastructure as underdeveloped as the US. Most of Europe has fibre-to-the-premise, and all of Europe has consumer internet packages that are faster and cheaper than you’re gonna find anywhere in the U.S.

There's (almost) no FTTH in Germany. The US used to be as bad as Germany, but it has improved significantly and is actually pretty decent these days (though connection speed is unevenly distributed).

Both countries are behind e.g. Sweden or Russia, but Germany by a much larger margin.

There's some trickery done in official statistics (e.g. by factoring in private connections that are unavailable to consumers) to make this seem better than it is, but ask anyone who lives there and you'll be surprised.

Re: We shrunk our Javascript monorepo git size

#195

Post by [deleted] »

wavemode wrote: post #181
crazygringo wrote: post #174

Earlier quoted context omitted.

> all while still arguing monorepos are easy and great I don't know anyone who says monorepos are easy. To the contrary, the tooling is precisely the hard part. But the point is that the difficulty of the tooling is a lot less than the difficulty of managing compatibility conflicts between tons of separate repos. Each esoteric bug in C only needs to be fixed once. Whereas your version compatibility conflict this week…

At Amazon, there is no monorepo. And the tooling to handle this is not even particularly conceptually complicated - a "versionset" is a set of versions - a set of pointers to a particular commit of a repository. When you build and deploy an application, what you're building is a versionset containing the correct versions of all its dependencies. And pull requests can span across multiple repositories. Working at Amaz…

[deleted]

Re: We shrunk our Javascript monorepo git size

#196

Post by nixosbestos »

dijit wrote: post #127
acdha wrote: post #126

Earlier quoted context omitted.

The extend phase starts when they make extensions which only work in their proprietary version. Putting extensive work into contributing them back is not the same.

Ok. There are a dozen examples of exactly this behaviour, and exactly this argumentation in response over the years. Right now the most important thing for them is for people to start thinking the microsoft fork is the superior one, even if things are “backported”.

No examples offered. And zero that I know of with respect to Git. This is how all open source development is done with big features - iterated on in a fork and proposed and merged in.

There are so many good things to criticize Microsoft for. When this is what people come with, it serves as a single of emotion-based ignorance and to ignore.

Re: We shrunk our Javascript monorepo git size

#198

Post by holowoodman »

immibis wrote: post #119
holowoodman wrote: post #69

Earlier quoted context omitted.

Germany. Deutsche Telekom is the former monopoly that was half-privatized around 1995 or something. The state still owns quite a large stake of it. They milk their ancient copper crap for everything they can while keeping prices high. They are refusing useful backbone interconnects to monopolize access to their customers (Actually they are not allowed to refuse. They just offer interconnections only in their data cen…

Wait, I live in that area. Does that mean I'm allowed to lay my own fiber into their open ditches too, or do they have special rights no one else has?

Afaik the special right is granted to everyone providing fibre services to the public to be informed about any ditches on public ground being dug and getting the opportunity to throw their fibre in before the ditch is closed again.

Re: We shrunk our Javascript monorepo git size

#199

Post by thiht »

haolez wrote: post #164
PoignardAzur wrote: post #90

Earlier quoted context omitted.

Oh for crying out loud. "EEE" isn't a magic incantation, it's the name of an actual policy with actual tangible steps that their executives were implementing back when the CEO thought open source was the greatest threat to their business model. Microsoft contributing to a project doesn't automatically make it EEE. For one thing, EEE was about adopting open standards in proprietary software. Microsoft during EEE didn'…

Well, most of their extensions to VSCode are proprietary. When their dominance in software development becomes irreversible, it's obvious that they will close things down and create new sources of income. The incentives are clear.

So what? You can use VSCodium and the OpenVSX marketplace if you like, no one is stopping you. It DOES mean you won’t be able to use some extensions that are published exclusively on the VSCode marketplace but guess what? You’re not entitled to every extension being accessible from all the stores, and you’re even less entitled to demand that all extensions are open source.

If Microsoft want to develop some proprietary extensions for VSCode it’s fine, everyone has this right. It has nothing to do with EEE.

Re: We shrunk our Javascript monorepo git size

#200

Post by jasonm23 »

mb7733 wrote: post #185
haolez wrote: post #164

Earlier quoted context omitted.

Well, most of their extensions to VSCode are proprietary. When their dominance in software development becomes irreversible, it's obvious that they will close things down and create new sources of income. The incentives are clear.

VSCode is _their_ product. It doesn't make sense to say that they are EEEing their own product. EEE is when you take some existing open standard, support it in proprietary a product, and then extend it in proprietary ways, thereby taking over the standard. It doesn't apply for a product that you originally created.

.... Fork not an original creation

that's how effective eee is, you don't know (or likely care) where MS ripped all this code.

This is not an accident. It's the point.

Post reply on HN