Reformatting 100k Files at Google in 2011 • Hacker News Board

Re: Reformatting 100k Files at Google in 2011

#11

Post by autarch » Sun, Jun 16, 2024, 2:09 AM UTC

wonger_ wrote: post #2

Autoformatting is so nice. Crazy to think that formatters only became popular after `gofmt`. I also found this related quote from Russ Cox intriguing: "Most people think that we format Go code with gofmt to make code look nicer or to end debates among team members about program layout. But the most important reason for gofmt is that if an algorithm defines how Go source code is formatted, then programs, like goimport…

The Perl world had perltidy (first release in 2002) many years before Go was even a thing. It's funny that Perl, a language notorious for it's "There's More Than One Way To Do It" (TMTOWTDI) philosophy had a tidier so early. Of course, perltidy is _ridiculously_ configurable.

One thing I really love about gofmt is that it has no configuration at all. I think that was a major "innovation" and I'd love to see more languages adopt this approach.

Re: Reformatting 100k Files at Google in 2011

#12

Post by flymasterv » Sun, Jun 16, 2024, 2:20 AM UTC

I don’t understand why they had to format 100k files. You enforce the format with a presubmit and let the code get formatted in the next change.

I have long felt that Google’s strength has always been making a bad architectural choice and then executing on it flawlessly. So many systems are designed in ways that require incredible technical execution to make them workable, and they do it.

Re: Reformatting 100k Files at Google in 2011

#13

Post by oceanstone » Sun, Jun 16, 2024, 2:23 AM UTC

flymasterv wrote: post #12

I don’t understand why they had to format 100k files. You enforce the format with a presubmit and let the code get formatted in the next change. I have long felt that Google’s strength has always been making a bad architectural choice and then executing on it flawlessly. So many systems are designed in ways that require incredible technical execution to make them workable, and they do it.

Ex: Angular

Re: Reformatting 100k Files at Google in 2011

#14

Post by jsolson » Sun, Jun 16, 2024, 2:28 AM UTC

vitiral wrote: post #7

I'm going to take a contrarian view here. Code formatting is amazing in a corporate environment where nobody truly cares about their code -- it's just a means to get a paycheck. It's also great for beginners to a language who are still trying to get a handle of the syntax. But where you are nearly the sole owner of a small library and you are crafting that library to be beautiful and understandable... there is someth…

[flagged]

Re: Reformatting 100k Files at Google in 2011

#15

Post by valicord » Sun, Jun 16, 2024, 2:29 AM UTC

flymasterv wrote: post #12

I don’t understand why they had to format 100k files. You enforce the format with a presubmit and let the code get formatted in the next change. I have long felt that Google’s strength has always been making a bad architectural choice and then executing on it flawlessly. So many systems are designed in ways that require incredible technical execution to make them workable, and they do it.

Because then every commit from now on has a ton of formatting changes that make it harder to see what was actually changed.

Re: Reformatting 100k Files at Google in 2011

#16

Post by rsc » Sun, Jun 16, 2024, 2:30 AM UTC

flymasterv wrote: post #12

I don’t understand why they had to format 100k files. You enforce the format with a presubmit and let the code get formatted in the next change. I have long felt that Google’s strength has always been making a bad architectural choice and then executing on it flawlessly. So many systems are designed in ways that require incredible technical execution to make them workable, and they do it.

If you do it that way, then what should be a 1-line BUILD file change turns into something that changes every line. It distracts from the actual purpose of the future change. Many directories aren't touched for long periods of time. A few months from now someone tries to make a 1-line change and is unpleasantly surprised they have to deal with tons of seemingly spurious formatting changes. Not good.

Putting the time of submitting the changes on a small team (mostly me, with approvals from Rob and help from Laurent) was absolutely the right tradeoff. It avoided the "unfunded mandate" and tech debt of making everyone else deal with it.

Update: I found the FAQ we wrote back then. It was very short. These were the last two questions:

Q: Who will update all the existing BUILD files?

A: We will. There are nearly 200,000 of them, and we’ll take care of that. We’re sending CLs out now. If you want to do it yourself, that’s fine: see go/buildifiernow for a tool that can help.

Q: You’re creating a lot more work for me.

A: We are creating significant amounts of work for ourselves, including reformatting all 193,000 BUILD files in google3. For the rest of the engineers in the company, we intend to make the transition as smooth as possible, with integration in Eclipse, Emacs, and Vim, as well as tools like Rosie and GenJsDeps. It is an explicit goal not to create significant work for other engineers. If, as we roll this out, you find that we’ve created noticeable work in your workflow, please let us know so that we can address that.

Re: Reformatting 100k Files at Google in 2011

#17

Post by lopkeny12ko » Sun, Jun 16, 2024, 2:30 AM UTC

flymasterv wrote: post #12

I don’t understand why they had to format 100k files. You enforce the format with a presubmit and let the code get formatted in the next change. I have long felt that Google’s strength has always been making a bad architectural choice and then executing on it flawlessly. So many systems are designed in ways that require incredible technical execution to make them workable, and they do it.

[flagged]

Re: Reformatting 100k Files at Google in 2011

#18

Post by refulgentis » Sun, Jun 16, 2024, 2:35 AM UTC

lopkeny12ko wrote: post #17

flymasterv wrote: post #12

I don’t understand why they had to format 100k files. You enforce the format with a presubmit and let the code get formatted in the next change. I have long felt that Google’s strength has always been making a bad architectural choice and then executing on it flawlessly. So many systems are designed in ways that require incredible technical execution to make them workable, and they do it.

[flagged]

It's saturday night, in spring, it's really beautiful out. Another year alive. It gives me joy and reminds me to ease off a bit.

In unrelated news, OP was suggesting no 100K file CL, and a presubmit. They were not disputing what the article said. They were suggesting sharding out the initial formatting change to 100K individual CLs.

Re: Reformatting 100k Files at Google in 2011

#19

Post by rsc » Sun, Jun 16, 2024, 2:38 AM UTC

My notes say it was 193k at the start. The final dashboard when we stopped said "216,626 / 216,890 = 99.8%; 264 to go".

The other correction I would make is that this post does not mention Nilton Volpato, who had written an earlier Buildifier and graciously accepted replacing his implementation with a new one and then taking over ownership for that new implementation as well. (Eventually ownership moved to Laurent's team.)

It looks like it was just under 2,000 commits. We did pretty extensive testing, by having Blaze load a BUILD file and its transitive closure and then dump that parsed form back out to a binary format. Any automated commit had to preserve that parsed-and-dumped binary format bit for bit. The slowest part of the testing was waiting for Blaze to do all the loads.

Every day I would prepare and test as many files as I could, break them into CLs (think PRs), mail Rob a shell script he could run to approve them all, and go to bed. Then I'd get up early in the morning (5am ET) to submit the changes, because there were various cached indexes that got updated when BUILD files got submitted, and it seemed better to send them when not many people would be working.

That scheme worked until a system did fall over and someone got paged, and then after that I agreed to only submit the large changes during business hours. :-)

Re: Reformatting 100k Files at Google in 2011

#20

Post by neilv » Sun, Jun 16, 2024, 2:40 AM UTC

vitiral wrote: post #7

I'm going to take a contrarian view here. Code formatting is amazing in a corporate environment where nobody truly cares about their code -- it's just a means to get a paycheck. It's also great for beginners to a language who are still trying to get a handle of the syntax. But where you are nearly the sole owner of a small library and you are crafting that library to be beautiful and understandable... there is someth…

Also, sometimes the formatting simply makes reading something messy merely tractable, not aesthetically pleasing.

Once, a product launch depended on me urgently kludging a device driver in Python (long story). And this involved a large hand-maintained mapping table. I wrote it quickly but carefully, and found some formatting that made the table readable enough, without implementing a minilanguage in Python.

But the Black formatter had been rigged to run automatically on commit, so... poof! :)