Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reformatting 100k Files at Google in 2011

laurent.le-brun.eu

Post reply on HN
Print view View original

61–70 of 162 posts

Re: Reformatting 100k Files at Google in 2011

#61

Post by rsc »

randomifcpfan wrote: post #59
lopkeny12ko wrote: post #56

Earlier quoted context omitted.

What is build_cleaner?

A tool for updating bazel build target dependencies. It inspects build files and source code, then adds/removes dependencies from build targets as needed. It requires using global include paths in C/C++ sources. It is not perfect, but it is pretty nice!

If you're using Go with Bazel, gazelle is available outside Google: https://github.com/bazelbuild/bazel-gazelle

Enabling tools like these was exactly the point of the enforced formatting. It worked extremely well.

Re: Reformatting 100k Files at Google in 2011

#62

Post by tengbretson »

afavour wrote: post #23
wonger_ wrote: post #2

Autoformatting is so nice. Crazy to think that formatters only became popular after `gofmt`. I also found this related quote from Russ Cox intriguing: "Most people think that we format Go code with gofmt to make code look nicer or to end debates among team members about program layout. But the most important reason for gofmt is that if an algorithm defines how Go source code is formatted, then programs, like goimport…

I’ve wondered before whether the world would be well served by a programming language (or source control system, I suppose) that just stores ASTs in files rather than text code. When users open the file the editor formats to whatever their personal preference is, then saves edits back to the AST. It really is dumb to be arguing over tabs vs spaces, after all.

Does that mean that a file with a syntax error is not able to be saved?

Re: Reformatting 100k Files at Google in 2011

#63

Post by SR2Z »

tengbretson wrote: post #62
afavour wrote: post #23

Earlier quoted context omitted.

I’ve wondered before whether the world would be well served by a programming language (or source control system, I suppose) that just stores ASTs in files rather than text code. When users open the file the editor formats to whatever their personal preference is, then saves edits back to the AST. It really is dumb to be arguing over tabs vs spaces, after all.

Does that mean that a file with a syntax error is not able to be saved?

Well, not as an AST at least. Presumably it would still be ok as text.

Re: Reformatting 100k Files at Google in 2011

#65

Post by tylerhou »

dmazzoni wrote: post #44
bsmith0 wrote: post #32

I'd be curious what role "global approvers" at Google/Google's scale typically have/how many are there/what's the process?

I was at Google until a few years ago. The purpose of global approvers was exactly things like this. If you want to do a mechanical change to an insanely huge number of files, they can potentially approve it. In my experience, global approvers were used extremely rarely, only in cases like this where the transformation was purely mechanical and it was possible to verify that there were no logic changes. Most of the t…

Even if you get global approval it is still good to split CLs to avoid e.g. merge conflicts.

Re: Reformatting 100k Files at Google in 2011

#66

Post by Arainach »

rapfaria wrote: post #64

After reformatting, did git blame always pointed to those commits and the most recent author, or were they added to --ignore-rev?

Google doesn't use Git internally, and its code search and source control tools expose the concept of "show blame before this change", so in practice reformats like this aren't troublesome with regard to blame.

Re: Reformatting 100k Files at Google in 2011

#67

Post by haburka »

rsc wrote: post #19

My notes say it was 193k at the start. The final dashboard when we stopped said "216,626 / 216,890 = 99.8%; 264 to go". The other correction I would make is that this post does not mention Nilton Volpato, who had written an earlier Buildifier and graciously accepted replacing his implementation with a new one and then taking over ownership for that new implementation as well. (Eventually ownership moved to Laurent's…

You didn’t have Rosie to automatically split up your changes and send them out yet?? That must have been rough. LSCs are way easier now

Re: Reformatting 100k Files at Google in 2011

#68

Post by __s »

tengbretson wrote: post #62
afavour wrote: post #23

Earlier quoted context omitted.

I’ve wondered before whether the world would be well served by a programming language (or source control system, I suppose) that just stores ASTs in files rather than text code. When users open the file the editor formats to whatever their personal preference is, then saves edits back to the AST. It really is dumb to be arguing over tabs vs spaces, after all.

Does that mean that a file with a syntax error is not able to be saved?

language servers have pushed things to inspecting source code while programmer has partially written code, so having a kind of (invalid-span content="garbage") node in AST helps

Re: Reformatting 100k Files at Google in 2011

#69

Post by duped »

__s wrote: post #68
tengbretson wrote: post #62

Earlier quoted context omitted.

Does that mean that a file with a syntax error is not able to be saved?

language servers have pushed things to inspecting source code while programmer has partially written code, so having a kind of (invalid-span content="garbage") node in AST helps

Language servers usually are designed around full/concrete syntax trees instead of ASTs for exactly this reason. Adding error nodes to the AST is a hack that hurts more than helps.

More technically, language servers usually have a CST that they use to build the AST incrementally, and the AST contains references back to the CST that generated it. This is what allows you to handle incremental text edits and compile small deltas to the AST instead of the typical batch compiler design that attempts to parse everything all at once.

Re: Reformatting 100k Files at Google in 2011

#70

Post by duped »

afavour wrote: post #23
wonger_ wrote: post #2

Autoformatting is so nice. Crazy to think that formatters only became popular after `gofmt`. I also found this related quote from Russ Cox intriguing: "Most people think that we format Go code with gofmt to make code look nicer or to end debates among team members about program layout. But the most important reason for gofmt is that if an algorithm defines how Go source code is formatted, then programs, like goimport…

I’ve wondered before whether the world would be well served by a programming language (or source control system, I suppose) that just stores ASTs in files rather than text code. When users open the file the editor formats to whatever their personal preference is, then saves edits back to the AST. It really is dumb to be arguing over tabs vs spaces, after all.

One very nice property of an AST is that it's an IR that is allowed be instable even if the syntax it represents is stabilized. If the AST becomes the source of truth on the file system you lose that property.

On top of that, now you need to write a parser and compiler for your AST file. It's probably very simple and does rudimentary validation, but that defeats the point of the AST - it's a valid, canonical representation of a program by construction.

All in all, it seems like a good idea, and people have done it. But there's also good reasons to be apprehensive.

And at the end of the day, you need to ingest text as input, and you need to do it as fast as possible. There's not a ton of benefit to keeping an AST around on disk and in sync with the text that generated it when you are already able to compute it faster than you can read and deserialize it.

Post reply on HN