Live data from Hacker News

Formatting a 25M-line codebase overnight

stripe.dev

81–90 of 115 posts

Re: Formatting a 25M-line codebase overnight

#81
post #7

I’m shocked at the 25M line part! That is a completely unfathomable amount of code for one codebase. I really want to know more about that.

Unless I’m mistaken, it’s a monorepo. So it’s not 25M LoC in a single app, it’s (all?) of their server-side code and shared libraries. There’s also a variety of other languages in use.

16 years and thousands of engineers write a lot of code.

Re: Formatting a 25M-line codebase overnight

#82
post #53

Earlier quoted context omitted.

I am more shocked by the "overnight" aspect. I tried running clang-format on the Chromium source (68,281 .cc files, 21 million lines according to wc): $ find chromium-149.0.7826.1/ -name " .cc" -exec cat {} + | wc 21640925 55715244 833460441 And that took less than 6 minutes on a single E5-2696 v3 from 2014: $ time find chromium-149.0.7826.1/ -name *.cc | parallel -j 16 clang-format $x>/dev/null real 0m5.666s user 1m…

My guess would be tooling. I think the Ruby formatters are written in Ruby. I’d guess the clang one is written in C.

Nah the article says it's rust and calling into a C library for parsing.

Re: Formatting a 25M-line codebase overnight

#83
post #70
post #66

I don't understand why the felt the need to do a big-bang merge like this. Its a formatter, so the files should be functionally equivalent before and after. Why not just enable it for new files/edit files for a while, then once comfortable apply it to old files in batches? What advantage does the big bang merge give? Seems higher risk for the same reward

could introduce subtle bugs, so doing it all at once while it's on the front of everybody's mind with as much comprehensive review and testing of parts or the whole to everybody's satisfaction. if you don't do it all at once, you'd need to repeat the same amount of testing multiple times. > files should be functionally equivalent before and after when you say something like this, the road you are on is paved with goo…

You can also introduce subtle bugs in your own feature development, and if you change formatting there it's also at the front of your mind.

I think the main argument for doing a big bang rewrite is that you have a defined before/after, otherwise you're stuck into an endless in-between.

Re: Formatting a 25M-line codebase overnight

#84

One of my first jobs was a small software company writing software for a small number of clients, in MS basic PDS. The lead developer didn't like to bother with formatting code, so I wrote a tool called makenice to format his nasty spaghetti gibberish into something with good indents and layout to make it easier for us normal people to parse. He was furious, literally spun in circles about it right in the office in f…

This kind of passive-aggressive bullshit is exactly what's wrong with tech. People don't decide things: they just passively resist, and authority ends up being a muddle of truncated information flows.

If someone sharing an old war story with what I felt was a positive and joking tone triggered you this bad, I feel bad for anyone working with you.

Nothing in tech is worth going through life miserable. Nothing.

Re: Formatting a 25M-line codebase overnight

#85
post #26

I'm surprised they went with a all-at-once reformat. Even when doing it over a weekend this is bound to mess with a lot of open PRs at their scale. I had to introduce a formatter in a few sizeable codebases in the past (few 100k to few million LOC), and I always did it incrementally via a script that reformatted all files that are not touched in any open PR. The initial run reformatted 95% of all files. Then I ran th…

both options have their pros and cons. if you utilize some form of ratcheting[1], you can sneak it in without your team knowing.. but all of your PRs for the foreseeable future will have a ton of reformatting screwing with your git blame. if you do it all at once, someone will have to sort out conflicts, but you can utilize `blame.ignoreRevsFile`[2] so that your history remains useful [1] https://github.com/diffplug/…

Even if you spread out reformatting over multiple commits in different PRs, you can still make use of blame.ignoreRevsFile as long as your pull request workflow doesn't enforce squash merges even when somebody took extra care to produce a nice commit history.

Re: Formatting a 25M-line codebase overnight

#88
post #56

One of my first jobs was a small software company writing software for a small number of clients, in MS basic PDS. The lead developer didn't like to bother with formatting code, so I wrote a tool called makenice to format his nasty spaghetti gibberish into something with good indents and layout to make it easier for us normal people to parse. He was furious, literally spun in circles about it right in the office in f…

reminds me of rob pike mentioning gofmt's style is "no one's favorite"

Having K&R brackets be a syntactical requirement and everything else is a syntax error is okay with me though.

Re: Formatting a 25M-line codebase overnight

#89
post #47

> We chose a Saturday to format the entire codebase to avoid merge conflicts. And while our test suite gave us high confidence we'd gotten everything right, it's always a bit daunting to have a diff so large that GitHub can't render it. The dart formatter has an internal sanity check. It walks through the unformatted and formatted strings in parallel skipping any whitespace. If any non-whitespace characters don't mat…

I imagine a fancier version would be to compare the Abstract Syntax Trees.

I've always thought it would make sense for formatters to be baked into the toolchain so that they can reuse the language's parser (presumably exposed as a library) and then be implemented via parsing to AST and then formatted back out so that they're guaranteed to be correct and normalized. This doesn't seem to be how most formatters work in practice though, although I'm not sure if it's because of performance reasons or a lack of support for the parser being exposed in language toolchains.

Re: Formatting a 25M-line codebase overnight

#90

> We chose a Saturday to format the entire codebase to avoid merge conflicts. And while our test suite gave us high confidence we'd gotten everything right, it's always a bit daunting to have a diff so large that GitHub can't render it. The dart formatter has an internal sanity check. It walks through the unformatted and formatted strings in parallel skipping any whitespace. If any non-whitespace characters don't mat…

Strictly speaking that wouldn’t work, since a1 is different from a 1, for example.

I don't think they're trying to say that this is a sufficient test for correctness but a necessary one.
Post reply on HN