We Put Half a Million Files in One Git Repository, Here’s What We Learned • Hacker News Board

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#51

Post by Gigachad » Thu, Jun 16, 2022, 6:24 AM UTC

alexdowad wrote: post #25

Gigachad wrote: post #2

>just under 60 million lines of code in 2022. I'm always surprised at the fact that almost every product has more lines of code than the entire Linux repo. The scale of these products is astounding.

The natural tendency in almost any software is to keep adding and adding, while rarely throwing anything out. More features, more code, more supported platforms, more supported languages, more this, more that, more, more, more. If it was a physical product, you couldn't keep making it bigger and more complex ad infinitum, because making a physical thing bigger takes more material, and bounded physical resources would…

Canva is absolutely crushing the established design market with their product right now so I'd say having all those features is pretty important for a good tool these days.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#52

Post by milosmns » Thu, Jun 16, 2022, 6:26 AM UTC

qbasic_forever wrote: post #32

barbs wrote: post #28

I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file. Does anyone have any good resources for why and how best to implement a monorepo?

You kind of have to experience the worst of boths worlds to understand where and how each method works and breaks down. With multiple repos it's harder for teams to share code and collaborate. Each team has a repo that becomes a little fiefdom where they are oblivious to who is using their code and how they're using it. Suddenly they'll push out what they think is an innocuous refactor and inadvertently break core fu…

This is valid. But I think it would make sense to challenge this and even (try to) leave if product is dictating tech what to do with their repos... especially if there's a boss figure pushing imaginary deadlines without talking to the team beforehand. There's a job crisis now, I know, but millions of files were not checked in during 2021. We should do better when it comes to organizing how we work

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#53

Post by SenHeng » Thu, Jun 16, 2022, 6:26 AM UTC

jamesfinlayson wrote: post #42

SenHeng wrote: post #33

Earlier quoted context omitted.

Remember back when people recommended commiting node_modules into git?

Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within. It is gone now.

The days before lock files were a thing and 'it works on my machine!' was rampant.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#54

Post by auscompgeek » Thu, Jun 16, 2022, 6:27 AM UTC

37 wrote: post #6

Pretty interesting.... I don't know that much about git, but still fun to read. I guess the main takeaway is don't put all your eggs in one basket? Although it kinda seems like they are going to stick with the monorepo, ("Here’s how we solve them at Canva") Also, I looked up .xlf files and I still don't understand. It's xml, that part makes sense, but it's basically a config file? To tell what process to read which f…

.xlf isn't a config file format here. It refers to XLIFF, a standard for translation files. https://en.wikipedia.org/wiki/XLIFF

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#55

Post by Galanwe » Thu, Jun 16, 2022, 6:28 AM UTC

nsoonhui wrote: post #44

urban_winter wrote: post #39

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project bound…

One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.

Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#56

Post by haizzz » Thu, Jun 16, 2022, 6:33 AM UTC

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository.

The content of these translation files are snapshot in time aligned with the text in our product so simply removing them we would lose all the changes made to translations each time texts are changed.

Sorry about that, hope this clears it up!

Edit: for more information on translation and xlf files, we have another blog post all about them https://canvatechblog.com/how-to-design-in-every-language-at...

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#57

Post by milosmns » Thu, Jun 16, 2022, 6:33 AM UTC

duskwuff wrote: post #19

nindalf wrote: post #14

git status scales with the number of files in the repo. Ask HN - Are there any common git operations that scale with the number of commits?

Repository maintenance operations like "git gc" scale with the total number of objects in the repo.

I used to work at a company where our git repository would get so polluted with change-related junk that we had to call (many times literally call) BitBucket and ask them to run git clean up on the remote. Otherwise, fetch/pull and many operations on remote would take forever or often even fail completely due to timeout or lack of RAM.

It's Thursday and you wanna release a hotfix - but nope - all builds are failing because git can't cope with junk and your CI can't proceed. You then have to call BB instead and ask them to clean up, hoping they'll do it within the same day. Ah, exciting times.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#58

Post by Maxious » Thu, Jun 16, 2022, 6:34 AM UTC

tehlike wrote: post #35

Did anyone use scalar from Microsoft? https://github.com/microsoft/scalar

As mentioned in the blog "we are moving towards providing a known version of git", probably Microsoft Git which includes the scalar command but also upstreams many of the optimizations to git core https://github.com/microsoft/scalar#why-did-scalar-move

disclaimer: canva staff working on source control

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#59

Post by rtpg » Thu, Jun 16, 2022, 6:35 AM UTC

arinlen wrote: post #23

rsanheim wrote: post #20

It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…

> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.

One reason for doing this is that someone might be using a very small set of the monorepo, so having everything prebuilt means that in general things are very fast, without constantly having to rebuild chunks of your system.

This is, I think, very common with stuff like browsers, where you have artifact checkouts that basically include built stuff since otherwise you're sitting there compiling forever all the time.

Stuff like Bazel in theory helps with this, but tools that help with this are either super idiosyncratic about how they work (meaning hard to adopt) or outright don't work.

I mean personally I would find that pretty annoying and unclean but I like DAGs.

Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned

#60

Post by brutusborn » Thu, Jun 16, 2022, 6:35 AM UTC

syspec wrote: post #34

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

That bit surprised me also. I don't have much experience with companies of this size so I have no real idea; how many people would you expect to work on source control at a company of this size? I had a look at Linkedin, they have 4,345 employees.