Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Formatting a 25M-line codebase overnight

stripe.dev

Post reply on HN
Print view View original

41–50 of 115 posts

Re: Formatting a 25M-line codebase overnight

#41

Post by jrajav »

skydhash wrote: post #28
hobofan wrote: post #26

I'm surprised they went with a all-at-once reformat. Even when doing it over a weekend this is bound to mess with a lot of open PRs at their scale. I had to introduce a formatter in a few sizeable codebases in the past (few 100k to few million LOC), and I always did it incrementally via a script that reformatted all files that are not touched in any open PR. The initial run reformatted 95% of all files. Then I ran th…

You can always let the team know so that they can apply the formatter on their PR branch.

This is exactly the remedy to the PR issue. I've "lucked" into owning a Prettier formatting pass at two different places now, and did the same process at each - full pass on master, simple step-by-step process to follow to update any PR by running the format script.

Re: Formatting a 25M-line codebase overnight

#42

Post by hobofan »

skydhash wrote: post #28
hobofan wrote: post #26

I'm surprised they went with a all-at-once reformat. Even when doing it over a weekend this is bound to mess with a lot of open PRs at their scale. I had to introduce a formatter in a few sizeable codebases in the past (few 100k to few million LOC), and I always did it incrementally via a script that reformatted all files that are not touched in any open PR. The initial run reformatted 95% of all files. Then I ran th…

You can always let the team know so that they can apply the formatter on their PR branch.

In the smaller migrations I did I tried that, but some way or another a decent chunk of the people still managed to get stuck in merge/rebase conflicts. I would almost explicitly not recommend giving that advise to the teams.

My rough blueprint for introducing formatter or linter nowadys would be:

- Recorded knowledge share session around how to set up the tools for local use 1-2 weeks before the initial rollout, and outline how the process will take place

- On the day of the initial rollout send out a reminder + the recording again

- Do the initial PR

- Incrementally do the rest of the migration, and subscribe to the PRs that drag out the process

Re: Formatting a 25M-line codebase overnight

#43

Post by hobofan »

rileymichael wrote: post #40
hobofan wrote: post #26

I'm surprised they went with a all-at-once reformat. Even when doing it over a weekend this is bound to mess with a lot of open PRs at their scale. I had to introduce a formatter in a few sizeable codebases in the past (few 100k to few million LOC), and I always did it incrementally via a script that reformatted all files that are not touched in any open PR. The initial run reformatted 95% of all files. Then I ran th…

both options have their pros and cons. if you utilize some form of ratcheting[1], you can sneak it in without your team knowing.. but all of your PRs for the foreseeable future will have a ton of reformatting screwing with your git blame. if you do it all at once, someone will have to sort out conflicts, but you can utilize `blame.ignoreRevsFile`[2] so that your history remains useful [1] https://github.com/diffplug/…

Yes, that is a good point. This is also why I personally would recommend to let a central person/team handle the reformatting rather than sneaking it into every PR (- see my sibling comment). That way you can be in charge of having a uniform style of commit messages to make the reformat commits easy to identify and create a well kept ignoreRevsFile. I think that provides the best of both worlds.

Re: Formatting a 25M-line codebase overnight

#45

Post by Terr_ »

CrzyLngPwd wrote: post #20

One of my first jobs was a small software company writing software for a small number of clients, in MS basic PDS. The lead developer didn't like to bother with formatting code, so I wrote a tool called makenice to format his nasty spaghetti gibberish into something with good indents and layout to make it easier for us normal people to parse. He was furious, literally spun in circles about it right in the office in f…

I find a lot of these conflicts I can't resolve when everybody agrees that the pain of ugly/unnecessary diffs is greater than the pain of minor formatting disagreements.

Re: Formatting a 25M-line codebase overnight

#46

Post by throwaway041207 »

Imustaskforhelp wrote: post #13
sikozu wrote: post #5

Earlier quoted context omitted.

The systems have to be written in some kind of programming language, and I think Ruby is a perfectly fine choice.

Not denying that Ruby is a perfectly fine choice but within the article itself it says that Stripe runs the world's largest Ruby codebase so certainly it might be testing the constraints of the language. The thing I am interested is that I don't suppose that Stripe always had these many LOC's and so I would be curious to know if at any point as the codebase was increasing, were they looking at other new languages whi…

Stripe uses Sorbet which, in my experience, increases LOC.

Re: Formatting a 25M-line codebase overnight

#47

Post by Terr_ »

munificent wrote: post #29

> We chose a Saturday to format the entire codebase to avoid merge conflicts. And while our test suite gave us high confidence we'd gotten everything right, it's always a bit daunting to have a diff so large that GitHub can't render it. The dart formatter has an internal sanity check. It walks through the unformatted and formatted strings in parallel skipping any whitespace. If any non-whitespace characters don't mat…

I imagine a fancier version would be to compare the Abstract Syntax Trees.

Re: Formatting a 25M-line codebase overnight

#48

Post by BobbyTables2 »

rileymichael wrote: post #40
hobofan wrote: post #26

I'm surprised they went with a all-at-once reformat. Even when doing it over a weekend this is bound to mess with a lot of open PRs at their scale. I had to introduce a formatter in a few sizeable codebases in the past (few 100k to few million LOC), and I always did it incrementally via a script that reformatted all files that are not touched in any open PR. The initial run reformatted 95% of all files. Then I ran th…

both options have their pros and cons. if you utilize some form of ratcheting[1], you can sneak it in without your team knowing.. but all of your PRs for the foreseeable future will have a ton of reformatting screwing with your git blame. if you do it all at once, someone will have to sort out conflicts, but you can utilize `blame.ignoreRevsFile`[2] so that your history remains useful [1] https://github.com/diffplug/…

That’s a neat feature, thanks for sharing.

Unfortunately I find that code bases lacking auto formatting are often littered with non functional changes as developers temporarily instrument code, remove it, but leave whitespace changes behind.

In terms of tracking code changes, one really would have to rewrite the entire history with each commit reformatted.

Re: Formatting a 25M-line codebase overnight

#49

Post by deathanatos »

varun_ch wrote: post #7

I’m shocked at the 25M line part! That is a completely unfathomable amount of code for one codebase. I really want to know more about that.

My (much smaller than Stripe) company is well over 4.5M at this point, and the graph is very much exponential.

AI has been a huge problem here: the amount of code is just exploding. Quality of the produced code is another matter.

Re: Formatting a 25M-line codebase overnight

#50

Post by Groxx »

bruckie wrote: post #22
varun_ch wrote: post #7

I’m shocked at the 25M line part! That is a completely unfathomable amount of code for one codebase. I really want to know more about that.

Only 25 million? :) Google had billions a decade ago... https://research.google/pubs/why-google-stores-billions-of-l...

iirc they also vendor(ed) many of their dependencies, several layers deep, which still counts for "stores" though it's rather different than "wrote" / "maintains".
Post reply on HN