Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

121–130 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#121

Post by prmoustache »

haizzz wrote: post #56

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply rem…

Can't those xlf files be stored in a separate repo or in an object storage and let the build system fetch them from there?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#122

Post by lultimouomo »

onion2k wrote: post #10
wichert wrote: post #8

Earlier quoted context omitted.

Reading the article they are not generated files, but files that are never touched by developers. Translators will work with those files. I expect that for translators they have a different sparse checkout that only fetches .xlf files for their target languages.

xlf files are usually generated. They're XML. No one wants to write that by hand.

Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand.

I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in.

Translations files, even if they are not created with a plain text editor but with some other tool that handles the XML layer, are clearly not generated, as long as the translation is done by a human.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#123

Post by __alexs »

the_biot wrote: post #73

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

Yet another example of Additive Bias: https://brainlenses.substack.com/p/additive-bias

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#124

Post by onion2k »

lultimouomo wrote: post #122
onion2k wrote: post #10

Earlier quoted context omitted.

xlf files are usually generated. They're XML. No one wants to write that by hand.

Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand. I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in.…

For the app I work at the moment we use https://lokalise.com/. We add translation strings to a SaaS app, and then the translation team translate them. I've written a build tool that downloads the translation JSON files from the API using the CLI, or as part of our CI process. Other teams have tools that download their language packs for different iOS and Android apps. The translations are versioned in Lokalise and we using a branching strategy to manage the work. Lokalise has an option to generate xlf files (and JSON, xliff, arb, etc).

This is a very typical workflow. Most people are not out there modifying xlf files by opening them in a text editor. For a start, translations usually aren't done by developers.

(Huge shoutout to Lokalise btw. I can highly recommend it. It makes building a multi-lingual app across different platforms so much easier.)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#125

Post by _the_inflator »

ElemenoPicuares wrote: post #49
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Canva gives users without design tools expertise the ability to make fairly polished looking graphics with a super easy and intuitive interface. (As a designer, I can assure you that polished looking is not the same thing as designed.) It’s a very popular service, so they’re dealing with huge scale. Intuitive interfaces often come with complex mechanisms and lots of assets, and they have clients on every major mobile…

Don't rely on this anecdotal heuristic. Have a look at some enterprises. My experience: "How to solve scaling issues?" - "Automation? No, another team." ;)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#126

Post by GeneralMaximus »

haizzz wrote: post #66
zoomablemind wrote: post #29

> "...we found that .xlf files made up almost 70% of the total number of files. These .xlf files are generated... they are never manually edited by engineers..." First thought is why not to zip/tar away all of these "convenience" files per generation and add a line into build/install script to unpack them after checkout? Additionally, add the .xlf into .gitignore to exclude them from untracked. Noone cares to diff th…

I made another comment as well though tldr is these xlf files are translations tied to texts in code so we can't simply ignore them from the repository. The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc...

Could these translations be moved to another repository? Maybe they could be published as a separate NPM package that devs could install if they needed to look at the translations?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#127

Post by andi999 »

masklinn wrote: post #108
roflulz wrote: post #96

Earlier quoted context omitted.

monorepo until it becomes too big -> then, splitting it into 2-3 repos, until each one also gets too big to manage...

The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing…

But isnt it sort of correct to have translation data, which is text data, and there might be bug fixes, in the repo?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#128

Post by alephnan »

These engineering blogs are intended to promote the engineering culture of the company for recruitment process, and as a correlate an opportunity for the engineers to self-promote and boost their resume/promotion-worthiness internally.

The sentiment on this thread, if it is indicative of the greater talent pool, suggests this blog post is having the complete opposite effect.

I remember when Uber was proud of their thousands of repos. Here it’s the 60 million lines of code. It’s not just red flags, but seems like stuff that might get leaked to Programming Horrors / WTFs.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#130

Post by eatonphil »

SenHeng wrote: post #33
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Remember back when people recommended commiting node_modules into git?

Based on how brittle Github actions is I'd be ready to commit node_modules except for that I'm building cross-platform software with native dependencies.
Post reply on HN