Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The convergence of compilers, build systems and package managers

blog.ezyang.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: The convergence of compilers, build systems and package managers

#21

Post by thaumasiotes »

GFK_of_xmaspast wrote: post #15
henrikschroder wrote: post #5

Earlier quoted context omitted.

> I think AST and semantic-analyzer are going to play an increasing role in a variety of software development activities It would be fantastic if source control systems would work on the AST instead of the plain text files, so many annoying problems could be solved there.

What kinds of problems you talking about.

One really obvious one is merge conflicts in the following pattern:

    #old_file.py
    ...
    def func_before(*params):
        do_things()


    def func_after(*params):
        do_other_things()

    ...
If someone adds a function between func_before and func_after, and their coworker adds a different function also between func_before and func_after, you get a merge conflict because the line-based VCS doesn't know how the new functions should be ordered (or, potentially, interleaved ;) ). An AST-aware version control system could[1] realize that the order of function definitions is meaningless, and then wouldn't need to ask for help resolving the conflict.

[1] This isn't always true, so this might still involve some degree of being specialized-to-the-language. Or it might just mean that the AST the VCS worked with allows for fairly complicated types, and can distinguish between order-sensitive things and order-insensitive things.

Re: The convergence of compilers, build systems and package managers

#22

Post by glenjamin »

jimrandomh wrote: post #12

I'd prefer to see package managers be more separate. Downloading dependencies involves interacting with third parties and has security implications. That's fine when you're downloading software intentionally from a source you trust, but I've had the experience of downloading source code and noticing that the build system is downloading from sources I don't recognize or without https, and that's not so good.

I think this is a good argument for having the location of sources be configurable separately from the place where sources are included - but not necessarily in a different tool.

Re: The convergence of compilers, build systems and package managers

#23

Post by nwmcsween »

trishume wrote: post #3

This is a good analysis. A potential option for his proposed build output format is http://nixos.org/nix/ expressions. Nix is already kind of a fusion of a package manager and a build system. It's rather mature, has a decent ecosystem and does a lot of what he is looking for: - Complete determinism, handling of multiple versions, total dependency graphs - Parallel builds (using that dependency graph) - Distribution O…

Language specific tools (compiler in this case) could allow for things such as checking if library foo's api which library bar 'emulates' is sufficient for program c which depends on foo's API. With a lot more effort you could automate all dependency tracking and alternative library compatibility.

Re: The convergence of compilers, build systems and package managers

#24

Post by qznc »

johncolanduoni wrote: post #7

I think a good way to handle this convergence would be to move away from repeated invocations of the compiler. Instead, your build system/IDE could keep an active instance of the compiler running that keeps the last generated ASG/intermediate representation/machine code in memory, along with bookkeeping information that remembers the relations between these. When a file changes/a rebuild is requested, this lets the b…

There is certainly a trend to provide the frontend separately. C# with "Compiler as a Service", libclang, etc.

The reason is not just IDE integration. You want a separate lexer (and sometimes parser) for tools like go-fmt or good syntax highlighting. Static analyzers are getting more important and there are many.

Re: The convergence of compilers, build systems and package managers

#25

Post by iainmerrick »

GFK_of_xmaspast wrote: post #16
Tyr42 wrote: post #10

Earlier quoted context omitted.

"simple make" is ninja, right? https://ninja-build.org/ Already works with cmake and so on.

Doesn't work with fortran, which I found out to my chagrin just this afternoon.

Why doesn't it work? Ninja should work with any tools that you can run from the command line.

Re: The convergence of compilers, build systems and package managers

#26

Post by iainmerrick »

This is a great writeup of what I think is an unfortunate trend. When your compiler, build system and IDE are all tightly coupled, you end up locked into a single language.

It's hard to develop pieces of your code in multiple languages and have everything play well together. But for many projects that's a good way to do things. For example, in games programming, you might want to use an offline texture compression tool. Ideally that should be integrated into the overall build; but you shouldn't have to write your texture compressor in Haskell or C++ or whatever just because that's what the game is written in.

I think Xcode is what a lot of other IDEs and build systems are moving towards. Xcode is nice as long as you're working with normal code in permitted languages (C++, Obj-C, Swift) and permitted resource formats (NIBs). But if you need to do something slightly unusual, like calling a shell script to generate resources, it's horrible.

Oh, and I didn't even mention package managers! Having those tightly coupled to the other tools is horrible too.

Re: The convergence of compilers, build systems and package managers

#27

Post by bluetomcat »

iainmerrick wrote: post #26

This is a great writeup of what I think is an unfortunate trend. When your compiler, build system and IDE are all tightly coupled, you end up locked into a single language. It's hard to develop pieces of your code in multiple languages and have everything play well together. But for many projects that's a good way to do things. For example, in games programming, you might want to use an offline texture compression to…

> But if you need to do something slightly unusual, like calling a shell script to generate resources, it's horrible.

Not quite true. Xcode provides a "Run Script" build phase that lets you enter your shell script right into the IDE. A lot of handy environment variables are also there. You can easily reach your project via $SRCROOT, or modify the resources of the output bundle via "${CONFIGURATION_BUILD_DIR}/${PRODUCT_NAME}".

Re: The convergence of compilers, build systems and package managers

#28

Post by jacquesm »

In the days of Turbo Pascal even the editor was part of the whole. Compiler, build system, editor all tied together as a single executable. Package management wasn't on the horizon yet for that environment so it wasn't included. But there is definitely a precedent for this and this kind of convergence is hardly a new thing.

Personally I don't like it much. I prefer my tools to be separate and composable.

Re: The convergence of compilers, build systems and package managers

#29

Post by toolslive »

This is yet another problem caused by the fact compilers have a 'file in, file out' interface. (the other problem is performance of the compilation/linking/packaging process). There's simply no reason why input to a compiler should be a file, and no reason why the result should be one.

Re: The convergence of compilers, build systems and package managers

#30

Post by dgellow »

toolslive wrote: post #29

This is yet another problem caused by the fact compilers have a 'file in, file out' interface. (the other problem is performance of the compilation/linking/packaging process). There's simply no reason why input to a compiler should be a file, and no reason why the result should be one.

Could you be more explicit? What do you want instead of a file?
Post reply on HN