Really reminds me that there's nothing in principle stopping us from storing parse trees and exposing them via something git like so we can avoid even needing to format, let alone also needing to resolve a whole category of merge conflicts based on that formatting. I mean a format is just a theme over your data -- I mean code.
Formatting a 25M-line codebase overnight
71–80 of 115 posts
Re: Formatting a 25M-line codebase overnight
#72> We chose a Saturday to format the entire codebase to avoid merge conflicts. And while our test suite gave us high confidence we'd gotten everything right, it's always a bit daunting to have a diff so large that GitHub can't render it. The dart formatter has an internal sanity check. It walks through the unformatted and formatted strings in parallel skipping any whitespace. If any non-whitespace characters don't mat…
I imagine a fancier version would be to compare the Abstract Syntax Trees.
The only issue is then you're at the mercy of whatever parser your formatter uses to construct the AST
Re: Formatting a 25M-line codebase overnight
#73> Given that complexity, the hypothesis was simple: tackle the hardest syntax first and the rest will follow. Always nice to see. I've seen people fall into the trap of designing for the common case, not realizing most of the code will be to deal with the less common cases.
In another field I have heard it called going for the jugular; the vivid description helps get the point across nicely.
If you want to master something, you will have to know the hardest part. So just deal with that first and then everything else is easy, because you are dealing with it as somebody who has already mastered the domain.
Re: Formatting a 25M-line codebase overnight
#74Surely, it no longer needs to be human-readable, and the era of write-only code is finally upon us with the dawn of AI writing our mealtickets. Why bother formatting 25m lines of slop, and why is AI wasting tokens on making code look human-readable anyway?
Every LLM I have ever asked about this says they perform better when they receive pretty-printed code because it is easier to see structure and priorities. It has been an almost universal recommendation for me, and it makes sense since LLMs are just mimicking human expression.
Re: Formatting a 25M-line codebase overnight
#75Cool story. The treat at the end was fun as well, thank you!
Re: Formatting a 25M-line codebase overnight
#76Earlier quoted context omitted.
Only 25 million? :) Google had billions a decade ago... https://research.google/pubs/why-google-stores-billions-of-l...
iirc they also vendor(ed) many of their dependencies, several layers deep, which still counts for "stores" though it's rather different than "wrote" / "maintains".
Very true. It was still hundreds of millions of lines of first party code a decade ago, and could easily be over a billion at this point.
Re: Formatting a 25M-line codebase overnight
#77> We chose a Saturday to format the entire codebase to avoid merge conflicts. And while our test suite gave us high confidence we'd gotten everything right, it's always a bit daunting to have a diff so large that GitHub can't render it. The dart formatter has an internal sanity check. It walks through the unformatted and formatted strings in parallel skipping any whitespace. If any non-whitespace characters don't mat…
Strictly speaking that wouldn’t work, since a1 is different from a 1, for example.
Re: Formatting a 25M-line codebase overnight
#78Earlier quoted context omitted.
iirc they also vendor(ed) many of their dependencies, several layers deep, which still counts for "stores" though it's rather different than "wrote" / "maintains".
Very true. It was still hundreds of millions of lines of first party code a decade ago, and could easily be over a billion at this point.
Yeah, I can definitely believe that Google would break over a billion handwritten. It's a big company that has been around for a long time.
It's still absurd. But believable.
Re: Formatting a 25M-line codebase overnight
#79Earlier quoted context omitted.
I imagine a fancier version would be to compare the Abstract Syntax Trees.
The only issue is then you're at the mercy of whatever parser your formatter uses to construct the AST
Well, if any (common, non-hobby) parser is thrown off by the reformatting, then it's probably not a safe reformatting either way.
Re: Formatting a 25M-line codebase overnight
#80One of my first jobs was a small software company writing software for a small number of clients, in MS basic PDS. The lead developer didn't like to bother with formatting code, so I wrote a tool called makenice to format his nasty spaghetti gibberish into something with good indents and layout to make it easier for us normal people to parse. He was furious, literally spun in circles about it right in the office in f…
Outside of the naming - this is a perfectly sane thing to do for developer comfort and can usually be accomplished with simple transformations. There are often limitations (like manually added indentation/spacing for alignment) but as long as you're very intentional about what changes you'll allow and have a good understanding of the language it can be an extremely safe operation.
I think git’s naming is actually pretty reasonable: smudge (on checkout) & clean (on stage).