Live data from Hacker News

We shrunk our Javascript monorepo git size

jonathancreamer.com

61–70 of 226 posts

Re: We shrunk our Javascript monorepo git size

#61

> We work in a very large Javascript monorepo at Microsoft we colloquially call 1JS. I used to call it office.com.. Teams is the worst offender there. Even a website with a cryptominer on it runs faster than that junk.

Hey, I worked with Jonathan on 1JS a while ago (on a team, Excel).

Just a note OMR (the office monorepo) is a different (and actually much larger) monorepo than 1JS (which is big on its own)

To be fair I suspect a lot of the bloat in both originates from the amount of home grown tooling.

Re: We shrunk our Javascript monorepo git size

#63
post #32

For those wondering where this new git-survey command is, it's actually not in git.git yet! The author is using microsoft's git fork, they've added this new command just this summer: https://github.com/microsoft/git/pull/667

I assume full-name-hash and path-walk are also only in the fork as well (or in git HEAD)? Can't see them in the man pages, or in the 2.47 changelog.

Yep. Path-walk is currently pending review here: https://lore.kernel.org/all/pull.1813.git.1728396723.gitgitg...

It more or less replaces the --full-name-hash option (again a very good cover letter that explains the differences and pros/cons of each very well!)

Re: We shrunk our Javascript monorepo git size

#64

> For many reasons, that's just too big, we have folks in Europe that can't even clone the repo due to it's size. What's up with folks in Europe that they can't clone a big repo, but others can? Also it sounds like they still won't be able to clone, until the change is implemented on the server side? > This meant we were in many occasions just pushing the entire file again and again, which could be 10s of MBs per fil…

The repo is probably hosted on the west coast, meaning it has to cross the Atlantic whenever you clone it from Europe?

Re: We shrunk our Javascript monorepo git size

#65
post #12

> we have folks in Europe that can't even clone the repo due to it's size. What is it about Europe that makes it more difficult? That internet in Europe isn't as good? Actually, I have heard that some primary schools in Europe lack internet. My grandson's elementary school in rural California (population <10k) had internet as far back as 1998.

In most EU countries we have multi-gigabit internet (for cheap too). Current offers are around ~5 GBIT speeds for 20 bucks a month.

Not in the UK. Still on 80Mbit VDSL here.

Re: We shrunk our Javascript monorepo git size

#66
post #57

Earlier quoted context omitted.

Let's pretend you didn't write the last 2 sentences... first of all "internet in Europe" makes close to zero sense to argue about. The article just uses it as a shortcut to not start listing countries. I live in a country where I have 10Gbps full-duplex and I pay 50$ / month, in "Europe". The issue is that some countries have telecom lobbies which are still milking their copper networks. Then the "competition committ…

What european countries still dont have fiber? //Confused swede with 10G fiber all over the place. Writing from literally the countryside next to nowhere.

Germany, GP's situation smells like their policies.

Re: We shrunk our Javascript monorepo git size

#67
Paraphrasing meat of the article:

- When you have multiple files in the repo which have the same trailing 16 characters in the repo path, git may wrongly calculate deltas, mixing up between those files. In here they had multiple CHANGELOG.md files mixed up.

- So if those files are big and change often, you end up with massive deltas and inflated repo size.

- There's a new git option (in Microsoft git fork for now) and config to use full file path to calculate those deltas, which fixes the issue when pushing, and locally repacking the repo.

```

git repack -adf --path-walk

git config --global pack.usePathWalk true

```

- According to a screenshot, Chromium repacked in this way shrinks from 100GB to 22GB.

- However AFAIU until GitHub enables it by default, GitHub clones from such repos will still be inflated.

Re: We shrunk our Javascript monorepo git size

#68

> we have folks in Europe that can't even clone the repo due to it's size. What is it about Europe that makes it more difficult? That internet in Europe isn't as good? Actually, I have heard that some primary schools in Europe lack internet. My grandson's elementary school in rural California (population <10k) had internet as far back as 1998.

The problem is probably that the repo is not hosted in Europe.

Re: We shrunk our Javascript monorepo git size

#69
post #57

Earlier quoted context omitted.

Let's pretend you didn't write the last 2 sentences... first of all "internet in Europe" makes close to zero sense to argue about. The article just uses it as a shortcut to not start listing countries. I live in a country where I have 10Gbps full-duplex and I pay 50$ / month, in "Europe". The issue is that some countries have telecom lobbies which are still milking their copper networks. Then the "competition committ…

What european countries still dont have fiber? //Confused swede with 10G fiber all over the place. Writing from literally the countryside next to nowhere.

Germany.

Deutsche Telekom is the former monopoly that was half-privatized around 1995 or something. The state still owns quite a large stake of it.

They milk their ancient copper crap for everything they can while keeping prices high.

They are refusing useful backbone interconnects to monopolize access to their customers (Actually they are not allowed to refuse. They just offer interconnections only in their data centers in the middle of nowhere, where you need to rent their (outrageously priced) rackspace and fibres because there is nothing else. They are refusing for decades to do anything useful at the big exchanges like DECIX).

And if there should ever be a small competitor that on their own tries to lay fibre somewhere, they quickly lay their own fibre into the open ditches (they are allowed to do that) and offer just enough rebates for their former copper customers to switch to their fibre that the competitor cannot recoup the invest and goes bankrupt. Since that dance is now known to everyone, even the announcement of Telekom laying their own fibres kills the competitors' projects there. So after a competitor's announcement of fibre rollout, Telekom does the same, project dead, no fibre rollout at all.

Oh, and since it is a partially-state-owned former monopoly/ministry, the state and competition authorities turn a blind eye to all that, when not actively promoting them...

Then there is the problem of "5G reception" vs. "5G reception with usable bandwidth". A lot of overbooking goes on, many cells don't have sufficient capacity allocated, so there are reports of 4G actually being faster in many places.

And also, yes, you can get 5G in a lot of actually populated areas. But you certainly will pay through the nose for that, usually you get a low-GB amount of traffic included, so maybe a tenth of the Microsoft monorepo in question. The rest is pay-10Eur-per-GB or something.

Re: We shrunk our Javascript monorepo git size

#70

> For many reasons, that's just too big, we have folks in Europe that can't even clone the repo due to it's size. What's up with folks in Europe that they can't clone a big repo, but others can? Also it sounds like they still won't be able to clone, until the change is implemented on the server side? > This meant we were in many occasions just pushing the entire file again and again, which could be 10s of MBs per fil…

> For many reasons, that's just too big, we have folks in Europe that can't even clone the repo due to it's size. I read that as an anecdote, a more complete sentence would be "We had a story where someone from Europe couldn't clone the whole repo on his laptop for him to use on a journey across Europe because his disk is full at the time. He has since cleared up the disk and able to clone the repo". I don't think it…

The European Union doesn't like when a file get too big and powerful. It needs to be broken apart in order to give smaller files a chance of success.
Post reply on HN