Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

181–190 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#181

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

[deleted]

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#182
post #61

Earlier quoted context omitted.

hey hey author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them

How do you split your XLIFF files? Does each project get one big one and the proliferation is simply due to number of languages, or do you have a more granular split (eg. if you've got one component, it will have dozens of XLIFF files for every language, instead of one per language)? By the numbers you mention, 70% of the files make a ratio of code files to translation files 1-3, so unless you only support 3-5 langua…

It's one XLIFF file per locale per component, not including the source en_US. We currently support 104 locales.

More info: https://news.ycombinator.com/item?id=28931601

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#183

Earlier quoted context omitted.

The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing…

> The problem is not the repository, it’s having the translation data in the repo. Does this data change as often as the code does? If not then get it out of the repo.

What if it changes more often than the code? Throw out the code?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#184

Earlier quoted context omitted.

CI/CD build the entire project. So if you make a breaking change in library the build will fail.

Sure, and what next? So considering that on big projects CI/CD may take up to an hour (in one of my projects it took 4 hours) the feedback loop would be great

One of the nice things about using Bazel is caching builds, avoiding rebuilding parts of the monorepo that are completely unaffected by someone's changes.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#185
post #120
post #118

Earlier quoted context omitted.

One of my biggest pet peeves: tracking generated files in version control. The only exception is our generated OpenAPI spec, because we want people to be explicit about modifying the API, and have a CI task that verifies that the API and OpenAPI spec match.

One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations. Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"

Hm... An interesting thought!

What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#186

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Not everyone has the skills to build the toolset and use it. My brother called last night to help him change some SASS variables in a bootstrap theme. He’s a data scientist and had no idea how to build bootstrap’s js and apply the new variables. If bootstrap came from npm fully built, over half of his problems he called me about (15 times!) would have been avoided.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#187

Earlier quoted context omitted.

> The main drawback is scalability of the tools involved like git. And if you can employ enough engineers to break git, you can probably afford a team to work on scaling git.

Git staring to break at 200-300 engineers pushing into it. Scaling git would take 200 more :)

Everything described in the post can be done by one engineer.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#188
post #68

Earlier quoted context omitted.

Author here, not everyone on the team works on git and git performance. Our team work on a variety of things that touch "code" in general like our ownership system and tooling, access control, bots and automation, etc...

Sorry, there is a quote from the article: "Kudos to other members of the Source Control team — Alex Sadleir, Wesley Li, Adam Murray, Matthew Chhoeu — who work on improving git performance at Canva" So I assume 5 people working specifically on Git performance ;-P

Heh, I can see how this can be misinterpreted :P

That's the entire Source Control team (of 5 people) today. Among other things, they work on improving Git performance ;)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#189

Earlier quoted context omitted.

CI/CD build the entire project. So if you make a breaking change in library the build will fail.

Sure, and what next? So considering that on big projects CI/CD may take up to an hour (in one of my projects it took 4 hours) the feedback loop would be great

> on big projects CI/CD may take up to an hour

No, it may not. Perhaps occasionally it does. That is a bug that you must fix - a pipeline that takes even 30 minutes is horrifically slow.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#190

These engineering blogs are intended to promote the engineering culture of the company for recruitment process, and as a correlate an opportunity for the engineers to self-promote and boost their resume/promotion-worthiness internally. The sentiment on this thread, if it is indicative of the greater talent pool, suggests this blog post is having the complete opposite effect. I remember when Uber was proud of their th…

[deleted]
Post reply on HN