Live data from Hacker News

The convergence of compilers, build systems and package managers

blog.ezyang.com

31–40 of 42 posts

Re: The convergence of compilers, build systems and package managers

#31

This is yet another problem caused by the fact compilers have a 'file in, file out' interface. (the other problem is performance of the compilation/linking/packaging process). There's simply no reason why input to a compiler should be a file, and no reason why the result should be one.

Can you give some examples of what we could pass to a compiler that is not a file ?

Re: The convergence of compilers, build systems and package managers

#32

Earlier quoted context omitted.

Doesn't work with fortran, which I found out to my chagrin just this afternoon.

Why doesn't it work? Ninja should work with any tools that you can run from the command line.

Things like module dependencies, apparently: https://groups.google.com/forum/#!searchin/ninja-build/fortr...

Re: The convergence of compilers, build systems and package managers

#33
post #30

This is yet another problem caused by the fact compilers have a 'file in, file out' interface. (the other problem is performance of the compilation/linking/packaging process). There's simply no reason why input to a compiler should be a file, and no reason why the result should be one.

Could you be more explicit? What do you want instead of a file?

Suppose the IDE has the code in memory, it wants it compiled (or syntax checked, or ...). Currently, it needs to write out a file, then call the compiler, and pick up the result from the file system, and parse the result. Ideal, would be a programmer's API to do this. But a compilation service, with RPC calls would already be a major improvement.

Also, if you store code, intermediate results, dependencies, ..., in a database (could even be in memory), you can reuse these intermediate results. The dependency graph can guide you to decide what needs to be changed etc.

Re: The convergence of compilers, build systems and package managers

#34
post #4

There is a problem but you're looking at it the wrong way. What's in the compiler's machine code generation phase that the build system needs to know about? If nothing, then making a monolithic system is only going to make your life miserable. Well-designed compilers are already split into (at least) two subsystems: frontend and backend. Frontend takes the program and spits an AST (very roughly speakign, although the…

> I think AST and semantic-analyzer are going to play an increasing role in a variety of software development activities It would be fantastic if source control systems would work on the AST instead of the plain text files, so many annoying problems could be solved there.

If version control stored refactorings (tree operations) against the AST it would open up a whole new world of possibilities.

Re: The convergence of compilers, build systems and package managers

#35
post #13

Earlier quoted context omitted.

> I think AST and semantic-analyzer are going to play an increasing role in a variety of software development activities It would be fantastic if source control systems would work on the AST instead of the plain text files, so many annoying problems could be solved there.

There are simpler and less intrusive ways to solve outside-AST issues (like an automatic reformatting pass before each build or each diff) What you're suggesting raises a bunch of new (non-trivial) issues: - What would you do with code comments? Things like "f(/+old_value+/new_value)". - How to store code before preprocessing (C and C++) ? - How to store files mixing several languages (PHP, HTML, Javascript) ? - How…

> - What would you do with code comments? Things like "f(/+old_value+/new_value)".

Comments are included in the AST, the AST should be reprojectable into canonical plaint text.

> - How to store code before preprocessing (C and C++) ?

This could get tricky, punt. cdata

> - How to store files mixing several languages (PHP, HTML, Javascript) ?

Same file format, different semantics. PHP is a DSL.

My new language manifesto includes having a mandatory publicly defined AST.

Re: The convergence of compilers, build systems and package managers

#36
post #30

Earlier quoted context omitted.

Could you be more explicit? What do you want instead of a file?

Suppose the IDE has the code in memory, it wants it compiled (or syntax checked, or ...). Currently, it needs to write out a file, then call the compiler, and pick up the result from the file system, and parse the result. Ideal, would be a programmer's API to do this. But a compilation service, with RPC calls would already be a major improvement. Also, if you store code, intermediate results, dependencies, ..., in a…

I'm not seeing the problem. You can already pipe inputs to the compiler, and those inputs don't have to be from a file on-disk, they can just be fed over stdin. Several compilers, including GCC, support using in-memory only movement of interstitial data between compilation phases instead of using intermediate temp files.

Re: The convergence of compilers, build systems and package managers

#38
post #35
post #13

Earlier quoted context omitted.

There are simpler and less intrusive ways to solve outside-AST issues (like an automatic reformatting pass before each build or each diff) What you're suggesting raises a bunch of new (non-trivial) issues: - What would you do with code comments? Things like "f(/+old_value+/new_value)". - How to store code before preprocessing (C and C++) ? - How to store files mixing several languages (PHP, HTML, Javascript) ? - How…

> - What would you do with code comments? Things like "f(/+old_value+/new_value)". Comments are included in the AST, the AST should be reprojectable into canonical plaint text. > - How to store code before preprocessing (C and C++) ? This could get tricky, punt. cdata > - How to store files mixing several languages (PHP, HTML, Javascript) ? Same file format, different semantics. PHP is a DSL. My new language manifest…

If it is reprojectable into canonical plain text, it's not really an AST - just an ST.

Re: The convergence of compilers, build systems and package managers

#39

This is a great writeup of what I think is an unfortunate trend. When your compiler, build system and IDE are all tightly coupled, you end up locked into a single language. It's hard to develop pieces of your code in multiple languages and have everything play well together. But for many projects that's a good way to do things. For example, in games programming, you might want to use an offline texture compression to…

> But if you need to do something slightly unusual, like calling a shell script to generate resources, it's horrible. Not quite true. Xcode provides a "Run Script" build phase that lets you enter your shell script right into the IDE. A lot of handy environment variables are also there. You can easily reach your project via $SRCROOT, or modify the resources of the output bundle via "${CONFIGURATION_BUILD_DIR}/${PRODUC…

That's the sort of stuff I mean when I say "horrible". :)

It'll just run the script every time, rather than doing anything smart with dependencies. Output from the script might or might not be picked up and tracked properly by the IDE. If you accidentally mess something up nothing will detect or prevent that.

(Edit: should add that I haven't given it a proper try in recent Xcode versions. I probably should.)

Re: The convergence of compilers, build systems and package managers

#40

Earlier quoted context omitted.

What kinds of problems you talking about.

One really obvious one is merge conflicts in the following pattern: #old_file.py ... def func_before(*params): do_things() def func_after(*params): do_other_things() ... If someone adds a function between func_before and func_after, and their coworker adds a different function also between func_before and func_after, you get a merge conflict because the line-based VCS doesn't know how the new functions should be orde…

Function definition order is certainly meaningful, at least to a human reader. Suppose co-worker 1's function is related to func_before, and co-worker 2's function is related to func_after, and the VCS flipped them around.

Or alternatively, I go in by myself and re-arrange the order of functions in a file to improve the clustering. If the VCS thinks the order of functions is meaningless, it won't recognize the change.

Post reply on HN