Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We shrunk our Javascript monorepo git size

jonathancreamer.com

Post reply on HN
Print view View original

181–190 of 226 posts

Re: We shrunk our Javascript monorepo git size

#181

Post by wavemode »

crazygringo wrote: post #174
smashedtoatoms wrote: post #150

Earlier quoted context omitted.

People act like managing lots of git repos is hard, then run into monorepo problems requiring them to fix esoteric bugs in C that have been in git for a decade, all while still arguing monorepos are easy and great and managing multiple repos is complicated and hard. It's like hammering a nail through your hand, and then buying a different hammer with a softer handle to make it hurt less.

> all while still arguing monorepos are easy and great I don't know anyone who says monorepos are easy. To the contrary, the tooling is precisely the hard part. But the point is that the difficulty of the tooling is a lot less than the difficulty of managing compatibility conflicts between tons of separate repos. Each esoteric bug in C only needs to be fixed once. Whereas your version compatibility conflict this week…

At Amazon, there is no monorepo.

And the tooling to handle this is not even particularly conceptually complicated - a "versionset" is a set of versions - a set of pointers to a particular commit of a repository. When you build and deploy an application, what you're building is a versionset containing the correct versions of all its dependencies. And pull requests can span across multiple repositories.

Working at Amazon had its annoyances, but dependency management across repos was not one of them.

Re: We shrunk our Javascript monorepo git size

#182

Post by clktmr »

atombender wrote: post #91
clktmr wrote: post #76

Earlier quoted context omitted.

[flagged]

Why? You do realize their fork is open source? The fix described in this post have been submitted as a patch to the official Git project. The fix is improving a legitimate inefficiency in Git, and does nothing towards "embracing", "extending", or "extinguishing" anything.

Can you imagine their fork extending git with a feature which is incompatible to mainline git and then forcing user's to switch to their fork via github? I can, and it will give them the power to extinguish mainline git and force everything they want on their users (telemetry, licence agreements, online registration...). That might be the reason they're embracing git right now. The fork being open source doesn't help at all.

I'm not saying this shouldn't be merged, but I think people should be aware and see the early signs.

Re: We shrunk our Javascript monorepo git size

#183

Post by jakub_g »

deskr wrote: post #147
jakub_g wrote: post #67

Paraphrasing meat of the article: - When you have multiple files in the repo which have the same trailing 16 characters in the repo path, git may wrongly calculate deltas, mixing up between those files. In here they had multiple CHANGELOG.md files mixed up. - So if those files are big and change often, you end up with massive deltas and inflated repo size. - There's a new git option (in Microsoft git fork for now) an…

> in Microsoft git fork for now Wait, what? Has MS forked git?

MS has had their fork of git for years, and they contributed many performance features for monorepos since then to the mainline.

Re: We shrunk our Javascript monorepo git size

#184

Post by zelphirkalt »

singron wrote: post #148
zelphirkalt wrote: post #72

Earlier quoted context omitted.

If you really need it pointed out, take it from a German neighbor: Telekom is running some extortion scheme or so here. Oh we could have gotten fiber to our house already ... if we paid them 800+ Euro! So we rather stick with our 100MBits or so connection that is not fiber but copper. If the German state does not intervene here, or the practices of ISPs and whoever has the power to build fiber changes, we will for th…

Is 800 euros that bad? In the US, we were quoted $10k a few years back. Even if fiber is already at the road, $800 is probably a fair price just to trench the line from the road to your home and install an entry point. If they provide free installation, then they have to make up the cost by raising your rates.

I think private households paying 800 Euro for what should be public infrastructure, being milked by ISPs is pretty bad.

Re: We shrunk our Javascript monorepo git size

#185

Post by mb7733 »

haolez wrote: post #164
PoignardAzur wrote: post #90

Earlier quoted context omitted.

Oh for crying out loud. "EEE" isn't a magic incantation, it's the name of an actual policy with actual tangible steps that their executives were implementing back when the CEO thought open source was the greatest threat to their business model. Microsoft contributing to a project doesn't automatically make it EEE. For one thing, EEE was about adopting open standards in proprietary software. Microsoft during EEE didn'…

Well, most of their extensions to VSCode are proprietary. When their dominance in software development becomes irreversible, it's obvious that they will close things down and create new sources of income. The incentives are clear.

VSCode is _their_ product. It doesn't make sense to say that they are EEEing their own product. EEE is when you take some existing open standard, support it in proprietary a product, and then extend it in proprietary ways, thereby taking over the standard. It doesn't apply for a product that you originally created.

Re: We shrunk our Javascript monorepo git size

#186

Post by avianlyric »

tazjin wrote: post #117
yunusabd wrote: post #10

> For many reasons, that's just too big, we have folks in Europe that can't even clone the repo due to it's size. What's up with folks in Europe that they can't clone a big repo, but others can? Also it sounds like they still won't be able to clone, until the change is implemented on the server side? > This meant we were in many occasions just pushing the entire file again and again, which could be 10s of MBs per fil…

> What's up with folks in Europe that they can't clone a big repo, but others can? They might be in a country with underdeveloped internet infrastructure, e.g. Germany))

I do t think there’s any country in Europe with internet infrastructure as underdeveloped as the US. Most of Europe has fibre-to-the-premise, and all of Europe has consumer internet packages that are faster and cheaper than you’re gonna find anywhere in the U.S.

Re: We shrunk our Javascript monorepo git size

#187

Post by hinkley »

acdha wrote: post #146
yunusabd wrote: post #10

> For many reasons, that's just too big, we have folks in Europe that can't even clone the repo due to it's size. What's up with folks in Europe that they can't clone a big repo, but others can? Also it sounds like they still won't be able to clone, until the change is implemented on the server side? > This meant we were in many occasions just pushing the entire file again and again, which could be 10s of MBs per fil…

My guess is that “Europe” is being used as a proxy for “high latency, low bandwidth” – especially if the person in question uses a VPN (especially one of those terrible “SSL VPN” kludges). It’s still surprisingly common to encounter software with poor latency handling or servers with broken window scaling because most of the people who work on them are relatively close and have high bandwidth connection.

Sounds, based on other responders, like high latency high bandwidth, which is a problem many of us have trouble wrapping our heads around. Maybe complicated by packet loss.

After COVID I had to set up a compressing proxy for Artifactory and file a bug with JFrog about it because some of my coworkers with packet loss were getting request timeouts that npm didn’t handle well at all. Npm of that era didn’t bother to check bytes received versus content-length and then would cache the wrong answer. One of my many, many complaints about what total garbage npm was prior to ~8 when the refactoring work first started paying dividends.

Re: We shrunk our Javascript monorepo git size

#188

Post by atombender »

clktmr wrote: post #182
atombender wrote: post #91

Earlier quoted context omitted.

Why? You do realize their fork is open source? The fix described in this post have been submitted as a patch to the official Git project. The fix is improving a legitimate inefficiency in Git, and does nothing towards "embracing", "extending", or "extinguishing" anything.

Can you imagine their fork extending git with a feature which is incompatible to mainline git and then forcing user's to switch to their fork via github? I can, and it will give them the power to extinguish mainline git and force everything they want on their users (telemetry, licence agreements, online registration...). That might be the reason they're embracing git right now. The fork being open source doesn't help…

Sure, I can imagine. But this isn't what's happening.

Re: We shrunk our Javascript monorepo git size

#190

Post by hinkley »

develatio wrote: post #74

Hacking Git sounds fun, but isn't there a way to just not have 2.500 packages in a monorepo?

Code line count tends to grow exponentially. The bigger the code base, the more unreasonable it is to expect people not to reinvent an existing wheel, due to ignorance of the code or fear of breaking what exists by altering it to handle your use case (ignorance of the uses of the code).

IME it takes less time to go from 100 modules to 200 than it takes to go from 50 to 100.

Post reply on HN