Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Put Half a Million Files in One Git Repository, Here’s What We Learned

canvatechblog.com

Post reply on HN
Print view View original

81–90 of 270 posts

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#81

Post by jdlshore »

syspec wrote: post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

They have “hundreds of developers.” It seems completely reasonable for them to have a team focused on DX. It’s one of my most common recommendations for rapidly-growing companies, which are often focused on features at the expense of productivity. A DX team is one of the lowest-hanging fruits for fixing the problems that causes.

(Improved testing practices is often second, but that’s not a low-hanging fruit. It’s a big, juicy fruit that’s waaaaay up in the canopy. And then comes joint product decision-making within the team, which may as well be on another planet entirely.)

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#82

Post by haizzz »

robertlagrant wrote: post #12
onion2k wrote: post #9

Earlier quoted context omitted.

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?

author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#83

Post by andi999 »

the_biot wrote: post #73

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

What would be a better structure?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#84

Post by victor9000 »

the_biot wrote: post #73

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's…

Yeah, this sounds more like a pile of technical debt attached to a competent business.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#87

Post by fomine3 »

haizzz wrote: post #56

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply rem…

Thanks for additional info. I'm happy to hear that localization is on your priority.

> English source strings used in our frontend code live in Typescript files (.messages.ts). Source strings used in our backend code live in Java files (Message.java). Our internationalization (i18n) pipeline converts this into a series of XLIFF files (ending in .xlf), with one file for each locale. All these files live in the repository, but the translated .xlf files should never be modified by hand since they are updated automatically when strings get translated.

Isn't it cache?

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#88

Post by ngshiheng »

nomilk wrote: post #65

> git status takes 10 seconds on average > running these commands multiple times a day reduces the total productive time engineers have every day I love the attention paid to this. Often opportunities to prioritise seemingly small efficiency gains are neglected. At 10 seconds per command, an engineer that uses git status 50 times per day spends ~10 minutes per day waiting; an entire work week per year!! Well above th…

I'm glad I'm not the only one that thinks that way. Even ~3s every git status is driving me insane now.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#89

Post by haizzz »

fomine3 wrote: post #87
haizzz wrote: post #56

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply rem…

Thanks for additional info. I'm happy to hear that localization is on your priority. > English source strings used in our frontend code live in Typescript files ( .messages.ts). Source strings used in our backend code live in Java files ( Message.java). Our internationalization (i18n) pipeline converts this into a series of XLIFF files (ending in .xlf), with one file for each locale. All these files live in the repos…

Not quite, the .messages.ts and Message.java files are only in English which are then converted into XLIFF files for other languages so they're the source of truth for what the string should be in different locales so they're not cache

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#90

Post by donw »

SenHeng wrote: post #33
rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

Remember back when people recommended commiting node_modules into git?

Pretty sure that recommendation came from a Git hosting service that charged by the megabyte.
Post reply on HN