Reformatting 100k Files at Google in 2011 • Hacker News Board

Re: Reformatting 100k Files at Google in 2011

#141

Post by neilv » Mon, Jun 17, 2024, 12:49 PM UTC

tczMUFlmoNk wrote: post #126

neilv wrote: post #20

Earlier quoted context omitted.

Also, sometimes the formatting simply makes reading something messy merely tractable, not aesthetically pleasing. Once, a product launch depended on me urgently kludging a device driver in Python (long story). And this involved a large hand-maintained mapping table. I wrote it quickly but carefully, and found some formatting that made the table readable enough, without implementing a minilanguage in Python. But the B…

Black supports ignoring sections with a `# fmt: off` directive, and a hand-formatted constant table is a common use for that: https://black.readthedocs.io/en/stable/usage_and_configurati... Other formatters have similar functionality; e.g.: - /* prettier-ignore */: https://prettier.io/docs/en/ignore.html#javascript - #[rustfmt::skip]: https://github.com/rust-lang/rustfmt?tab=readme-ov-file#tips

Thank you, that'll come in handy again, I'm sure.

Re: Reformatting 100k Files at Google in 2011

#142

Post by rsc » Mon, Jun 17, 2024, 1:38 PM UTC

haburka wrote: post #67

rsc wrote: post #19

My notes say it was 193k at the start. The final dashboard when we stopped said "216,626 / 216,890 = 99.8%; 264 to go". The other correction I would make is that this post does not mention Nilton Volpato, who had written an earlier Buildifier and graciously accepted replacing his implementation with a new one and then taking over ownership for that new implementation as well. (Eventually ownership moved to Laurent's…

You didn’t have Rosie to automatically split up your changes and send them out yet?? That must have been rough. LSCs are way easier now

(Rosie is a code janitor program at Google that takes a change that affects files all across the tree and automatically cuts it into individual changes that can be mailed out to the individual teams as CLs/PRs for their approval.)

Rosie existed but very much wanted to break up the CL into independent per-directory CLs, and since I was editing one file in every directory in the entire tree, that would have been 200,000 independent CLs. I broke the list up by top-level directory or sub-directory and hit 100+ directories at a time.

Rosie also really wants to run each affected directory's tests, and I did not, because at scale flaky tests and such would be a significant source of false positives. The bit-for-bit check on the internal parsed representation of the meaning of the BUILD file proved that the changes were no-ops. That was better than any tests of the code in the directory.

I was already automating everything else, including deciding which files to change, reverting edits in files that were concurrently modified (they got swept into the next attempt), and the testing. Running a shell command to actually make the CLs was not difficult. And it generating the approval script trivial too.

Rosie is great but it wasn't the right tool for this job.

Re: Reformatting 100k Files at Google in 2011

#143

Post by rsc » Mon, Jun 17, 2024, 1:39 PM UTC

vasco wrote: post #128

Sounds like a quick way to get Golang installed on every Google developer machine and drive adoption.

No, because Go compiles to binaries, like C/C++ but unlike Java or Python. Only the buildifier binary needed to be installed on everyone's machine.

(I think Go was already installed on every Google developer machine at that point anyway.)

Re: Reformatting 100k Files at Google in 2011

#144

Post by rsc » Mon, Jun 17, 2024, 1:40 PM UTC

fragmede wrote: post #135

jerf wrote: post #98

The term "bikeshedding" comes up a lot on HN, when people spend a lot of time spinning wheels in endless debates about the things that are easy to debate while indefinitely deferring the tough conversations, but I think an underappreciated aspect of this meme is that the unimportant conversations quite frequently really are just that; unimportant, to everybody involved. Often (but not always) nobody actually cares. C…

> Caring is not necessarily represented by the amount of time or text spewed forth on a topic. The meta says that it is. there are only 81600 seconds most days, and you get to choose them how you want, so choose how you spend them wisely. if that's arguing over tabs or spaces, then that's your choice.

This One Trick Will Add 80 Minutes To Your Day

Re: Reformatting 100k Files at Google in 2011

#145

Post by rsc » Mon, Jun 17, 2024, 1:44 PM UTC

rapfaria wrote: post #64

After reformatting, did git blame always pointed to those commits and the most recent author, or were they added to --ignore-rev?

Not git, but yes, it did mess up the blame lines. Luckily most people don't care about the attribution on BUILD files. But I did get the occasional email for years afterward about "hey looks like you added this build target, can you explain it?" or just an automated tool deciding I was the last person who edited the file so I was a good reviewer for a future change. Those pings were a fun tour of directories I'd never heard of (but did edit). Been a few years since I got one of those.

Re: Reformatting 100k Files at Google in 2011

#146

Post by rsc » Mon, Jun 17, 2024, 1:50 PM UTC

aiuto wrote: post #110

I want to start by saying that I do not want to diminish or disparage the work that Russ, Rob, Laurent, and others have done. It has made the Google code base better. That is an inarguable fact. Nor do I want to pick on buildifier or gofmt or any other tool as a singleton problem. I'll talk about buildifier because that is what I personally fight with. Others may have different demons. (YDMV - "your daemon may vary".…

I can't speak to what has happened with Buildifier, but in general you are right. It has to be a hard rule that if you change the format rules, you have to reformat everything to match.

If that means it's too hard to change the format rules, then don't change the format rules. And if you don't reformat, then it has to be a clear rule known to everyone (or written down somewhere you can point to) that incidental formatting changes are acceptable and not something you are allowed to push back on.

I can speak to Go and gofmt, and there we are VERY reluctant to change formatting rules. It does happen for the odd corner case once in a while, but nothing that would cause "changes far beyond my 2-line fix".

Re: Reformatting 100k Files at Google in 2011

#147

Post by jerf » Mon, Jun 17, 2024, 2:10 PM UTC

fragmede wrote: post #135

jerf wrote: post #98

The term "bikeshedding" comes up a lot on HN, when people spend a lot of time spinning wheels in endless debates about the things that are easy to debate while indefinitely deferring the tough conversations, but I think an underappreciated aspect of this meme is that the unimportant conversations quite frequently really are just that; unimportant, to everybody involved. Often (but not always) nobody actually cares. C…

> Caring is not necessarily represented by the amount of time or text spewed forth on a topic. The meta says that it is. there are only 81600 seconds most days, and you get to choose them how you want, so choose how you spend them wisely. if that's arguing over tabs or spaces, then that's your choice.

The meta would say that if people were optimally spending their time on things that matter to them. They don't. If they did bikeshedding wouldn't exist. It obviously does.

This is basically the same claim that economics can treat humans as perfectly rational actors perfectly rationally pursuing their perfectly rational goals. It is not a good model of humanity.

Re: Reformatting 100k Files at Google in 2011

#148

Post by freedomben » Mon, Jun 17, 2024, 2:42 PM UTC

autarch wrote: post #11

wonger_ wrote: post #2

Autoformatting is so nice. Crazy to think that formatters only became popular after `gofmt`. I also found this related quote from Russ Cox intriguing: "Most people think that we format Go code with gofmt to make code look nicer or to end debates among team members about program layout. But the most important reason for gofmt is that if an algorithm defines how Go source code is formatted, then programs, like goimport…

The Perl world had perltidy (first release in 2002) many years before Go was even a thing. It's funny that Perl, a language notorious for it's "There's More Than One Way To Do It" (TMTOWTDI) philosophy had a tidier so early. Of course, perltidy is _ridiculously_ configurable. One thing I really love about gofmt is that it has no configuration at all. I think that was a major "innovation" and I'd love to see more lang…

Perl is a fascinating example because it was so ahead of its time, so innovative, forward-thinking, so wonderful, and so loved, and yet today is looked upon with such disdain. I remember people battling for Perl positions and being so happy to get them! Now you can't even find people that admit to knowing Perl ;-)

Someday I want to study this and really understand what happened.

Re: Reformatting 100k Files at Google in 2011

#149

Post by freedomben » Mon, Jun 17, 2024, 2:50 PM UTC

rsc wrote: post #25

afavour wrote: post #23

Earlier quoted context omitted.

I’ve wondered before whether the world would be well served by a programming language (or source control system, I suppose) that just stores ASTs in files rather than text code. When users open the file the editor formats to whatever their personal preference is, then saves edits back to the AST. It really is dumb to be arguing over tabs vs spaces, after all.

At that point you could even use different languages. Maybe you like programs that look like Lisp and I don't. There was a project at Microsoft Research in the late 1990s/early 2000s that did exactly this - storing ASTs in source control instead of code - but the name escapes me at the moment.

That's a fascinating idea, but I wonder if code written in Go or similar language would make for some gargantuan and awful-to-read Lisp, and possibly vice-versa (Go with a ton of functions that are all only a few lines long. Sounds ok to me, but very different from any Go code I've worked on that I didn't write :-D).

Re: Reformatting 100k Files at Google in 2011

#150

Post by freedomben » Mon, Jun 17, 2024, 2:56 PM UTC

omoikane wrote: post #35

wonger_ wrote: post #2

Autoformatting is so nice. Crazy to think that formatters only became popular after `gofmt`. I also found this related quote from Russ Cox intriguing: "Most people think that we format Go code with gofmt to make code look nicer or to end debates among team members about program layout. But the most important reason for gofmt is that if an algorithm defines how Go source code is formatted, then programs, like goimport…

Autoformatters got popular was because a lot people don't care about formatting, and those who do care can't win against the auto part of autoformatters. It works for go because gofmt was there from the start, so even if you are returning a multi-dimension array and elements come out unaligned, that's just accepted as how it is and nobody cares. For other languages, people will have to either accept "not caring" as b…

Yes precisely this. I worked on a team in C++ that vertically aligned things, and at first it seemed like extra work but once I started seriously reading the code, I had an epiphany of realization that it is an amazing improvement in readability, but also makes it super easy to find certain bugs, and also do block editing in tools like vim. After leaving that team, I found that everywhere I tried to introduce this people just shrugged and rolled an auto-formatter over and told me that it's not worth thinking about.

IMHO autoformatters are awesome until there's something you care about that the implementers didn't care about, then they are horrible. Problem is, the people who put a lot of thought into those decisions are often in the minority and tend to lose the argument.