Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We shrunk our Javascript monorepo git size

jonathancreamer.com

Post reply on HN
Print view View original

41–50 of 226 posts

Re: We shrunk our Javascript monorepo git size

#41

Post by tux3 »

eviks wrote: post #7

upd: silly mistake - file name does not include its full path The explanation probably got lost among all the gifs, but the last 16 chars here are different: > was actually only checking the last 16 characters of a filename > For example, if you changed repo/packages/foo/CHANGELOG.md, when git was getting ready to do the push, it was generating a diff against repo/packages/bar/CHANGELOG.md!

Derrick provides a better explanation in this cover letter: https://lore.kernel.org/git/pull.1785.git.1725890210.gitgitg...

(See also the path-walk API cover letter: https://lore.kernel.org/all/pull.1786.git.1725935335.gitgitg...)

The example in the blog post isn't super clear, but Git was essentially taking all the versions of all the files in the repo, putting the last 16 bytes of the path (not filename) in a hash table, and using that to group what they expected to be different versions of the same file together for delta compression.

Indeed in the blog it doesn't work, because foo/CHANGELOG.md and bar/CHANGELOG.md is only 13 chars, but you have to imagine the paths have a longer common suffix. That part is fixed by the --full-name-hash option, now you compare the full path instead of just 16 bytes.

Then they talk about increasing the window size. That's kind of a hack to workaround bad file grouping, but it's not the real fix. You're still giving terrible inputs to the compressor and working around it by consuming huge amounts of memory. So that was a bit confusing to present it as the solution. The path walk API and/or --full-name-hash are the real interesting parts here =)

Re: We shrunk our Javascript monorepo git size

#43

Post by alexanderchr »

tinco wrote: post #36
wodenokoto wrote: post #25

Earlier quoted context omitted.

We were all impressed with google docs, but office.com is way more impressive. Collaborative editing between a web app, two mobile anpps and a desktop app with 30 years of backwards compatibility and it pretty much just works. No wonder that took a lot of JavaScript!

To be fair, we were impressed with Google Docs 15 years ago. Not saying office.com isn't impressive, but Google Docs certainly isn't impressive today. My company still uses GSuite, as I don't like being in Microsoft's ecosystem and we don't need any advanced features of our office suite but Google Docs and the rest of the GSuite seem to be intentionally held back to technology of the early 2010's.

Google docs certainly haven't changed much the last 5-10 years. I wonder if that's an intentional choice, or if it is because those that built it and understand how it works are long gone to work on other things.

Re: We shrunk our Javascript monorepo git size

#44

Post by barrkel »

jillesvangurp wrote: post #31
gnrlst wrote: post #12

Earlier quoted context omitted.

In most EU countries we have multi-gigabit internet (for cheap too). Current offers are around ~5 GBIT speeds for 20 bucks a month.

Sadly, I'm in Germany. Which is a third world country when it comes to decent connectivity. They are rolling out some fiber now in Berlin. Finally. But very slowly and not to my building any time soon. Most of the country is limited to DSL speeds. Mobile coverage is getting better but still non existent outside of cities. Germany has borders with nine countries. Each of those have better connectivity than Germany. I'…

When I travel in Germany I use a Deutsche Telekom pay as you go SIM in a 5G hotspot, and generally get about 200Mbit throughtput, which is far higher than you can expect any place you're staying to provide. It's €7 a day (or €100 a month) but it's worth it to avoid the terrible internet.

Re: We shrunk our Javascript monorepo git size

#45

Post by jonathanlydall »

issung wrote: post #19

Having someone in arms reach to help out that knows the inner workings of Git so much must be a lovely perk of working on such projects at companies of this scale.

Certainly being in an org which has close ties to entities like GitHub helps, but any team in any org with that number of developers can justify the cost of bringing in a highly specialized consultant to solve an almost niche problem like this.

Re: We shrunk our Javascript monorepo git size

#46

Post by aniviacat »

jillesvangurp wrote: post #31
gnrlst wrote: post #12

Earlier quoted context omitted.

In most EU countries we have multi-gigabit internet (for cheap too). Current offers are around ~5 GBIT speeds for 20 bucks a month.

Sadly, I'm in Germany. Which is a third world country when it comes to decent connectivity. They are rolling out some fiber now in Berlin. Finally. But very slowly and not to my building any time soon. Most of the country is limited to DSL speeds. Mobile coverage is getting better but still non existent outside of cities. Germany has borders with nine countries. Each of those have better connectivity than Germany. I'…

> Mobile coverage is getting better but still non existent outside of cities.

According to the Bundesnetzagentur over 90% [1] of Germany has 5G coverage (and almost all of the rest has 4G [2]).

[1] https://www.bundesnetzagentur.de/SharedDocs/Pressemitteilung...

[2] https://gigabitgrundbuch.bund.de/GIGA/DE/MobilfunkMonitoring...

Re: We shrunk our Javascript monorepo git size

#48

Post by lastdong »

tux3 wrote: post #41
eviks wrote: post #7

upd: silly mistake - file name does not include its full path The explanation probably got lost among all the gifs, but the last 16 chars here are different: > was actually only checking the last 16 characters of a filename > For example, if you changed repo/packages/foo/CHANGELOG.md, when git was getting ready to do the push, it was generating a diff against repo/packages/bar/CHANGELOG.md!

Derrick provides a better explanation in this cover letter: https://lore.kernel.org/git/pull.1785.git.1725890210.gitgitg... (See also the path-walk API cover letter: https://lore.kernel.org/all/pull.1786.git.1725935335.gitgitg... ) The example in the blog post isn't super clear, but Git was essentially taking all the versions of all the files in the repo, putting the last 16 bytes of the path (not filename) in a hash…

Thank you! I ended up having to look at the PR to make any sense of the blog post, but your explanation and links makes things much clearer

Re: We shrunk our Javascript monorepo git size

#49

Post by eviks »

daenney wrote: post #28
eviks wrote: post #7

upd: silly mistake - file name does not include its full path The explanation probably got lost among all the gifs, but the last 16 chars here are different: > was actually only checking the last 16 characters of a filename > For example, if you changed repo/packages/foo/CHANGELOG.md, when git was getting ready to do the push, it was generating a diff against repo/packages/bar/CHANGELOG.md!

File name doesn’t necessarily include the whole path. The last 16 characters of CHANGELOG.md is the full file name. If we interpret it that way, that also explains why the filepathwalk solution solves the problem. But if it’s really based on the last 16 characters of just the file name, not the whole path, then it feels like this problem should be a lot more common. At least in monorepos.

yes, this makes sense, thanks for pointing it out, silly confusion on my part

Re: We shrunk our Javascript monorepo git size

#50

Post by esperent »

wodenokoto wrote: post #25
bubblesnort wrote: post #15

> We work in a very large Javascript monorepo at Microsoft we colloquially call 1JS. I used to call it office.com.. Teams is the worst offender there. Even a website with a cryptominer on it runs faster than that junk.

We were all impressed with google docs, but office.com is way more impressive. Collaborative editing between a web app, two mobile anpps and a desktop app with 30 years of backwards compatibility and it pretty much just works. No wonder that took a lot of JavaScript!

We use MS Teams at my company. The Word and Excel in the Windows Teams app are so buggy that I can almost never successfully open a file. It just times out and eventually shows a "please try again later" message nearly every time. I've uninstalled and reinstalled the Teams app four or five times trying to fix this.

We've totally given up any kind of collaborative document editing because it's too frustrating, or we use Notion instead, which for all it's fault, at least the basic stuff like loading a bloody file works...

Post reply on HN