Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incremental Compilation

blog.rust-lang.org

Post reply on HN
Print view View original

31–40 of 74 posts

Re: Incremental Compilation

#31

Post by Retra »

dllthomas wrote: post #26
pcwalton wrote: post #18

Earlier quoted context omitted.

Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets. Not to mention that the size of your instance variables leaks into your public interface unless you manually heap allocate and use the pimpl idiom. In any case, Rust stores the interfaces to libraries in serialized binary form (including documentation, following a standardized form…

"C++ did it right" seems a separate position from that put forward by the parent comment. I definitely think there's benefit to being able to view interface as separate from implementation, although that might well be better supported by tooling (editor folding, documentation generation) than manual maintenance - which you seem to be doing well. I don't know whether there is benefit to being able to edit interface se…

Usually if you edit the interface, you also have to edit the implementation. The only time I've ever wanted to edit an interface alone is when there wasn't any implementation written yet, which makes it a moot point anyway.

Re: Incremental Compilation

#32

Post by skybrian »

pcwalton wrote: post #27
taneq wrote: post #25

Earlier quoted context omitted.

If you have a single source file that's big enough for the compiler to bog down on it, then either that compiler is beyond ridiculously slow, or you need to refactor.

The tricky part is the dependencies between files. This is what C++ uses header files for, but Rust doesn't have those.

Languages like Go and Java can do separate compilation by reading header information out of the object files of imported packages/classes.

Re: Incremental Compilation

#33

Post by flukus »

dllthomas wrote: post #26
pcwalton wrote: post #18

Earlier quoted context omitted.

Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets. Not to mention that the size of your instance variables leaks into your public interface unless you manually heap allocate and use the pimpl idiom. In any case, Rust stores the interfaces to libraries in serialized binary form (including documentation, following a standardized form…

"C++ did it right" seems a separate position from that put forward by the parent comment. I definitely think there's benefit to being able to view interface as separate from implementation, although that might well be better supported by tooling (editor folding, documentation generation) than manual maintenance - which you seem to be doing well. I don't know whether there is benefit to being able to edit interface se…

Two seconds of google found a .h file generator: http://www.radwin.org/michael/2011/05/10/stubgen/ . I'm sure there are others.

Re: Incremental Compilation

#34

Post by berkut »

dllthomas wrote: post #26
pcwalton wrote: post #18

Earlier quoted context omitted.

Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets. Not to mention that the size of your instance variables leaks into your public interface unless you manually heap allocate and use the pimpl idiom. In any case, Rust stores the interfaces to libraries in serialized binary form (including documentation, following a standardized form…

"C++ did it right" seems a separate position from that put forward by the parent comment. I definitely think there's benefit to being able to view interface as separate from implementation, although that might well be better supported by tooling (editor folding, documentation generation) than manual maintenance - which you seem to be doing well. I don't know whether there is benefit to being able to edit interface se…

Writing applications which have an API other people write plugins against is one: commercial apps ship the .h files, against which plugin-writers only have those and build plugins for the host apps.

Yes you still obviously have to keep them in sync, but the point is you (as a commercial software vendor) can ship just the headers and keep the internal implementation details secret and change how things work completely. Which is useful for commercial vendors of software.

Re: Incremental Compilation

#35

Post by pjmlp »

pcwalton wrote: post #6
skybrian wrote: post #3

Nice to see it. Incremental compilation is one of those things that production compilers have but research languages usually don't have. You only need it when you have a lot of code, but then you really need it.

Production compilers often don't have it either. (Having to split up your code manually into separate .cpp [or what have you] files doesn't qualify as incremental compilation—that's just separate compilation. Incremental compilation is when the compiler automatically figures out what needs to be recompiled, even when all you hand it is a big blob of code that hasn't been manually split up by a human in any way.)

Lucid C++ was already doing incremental compilation at function level for 1993 thanks their pivoting from Lisp machines.

If I recall correctly they demo it on this video

https://www.youtube.com/watch?v=pQQTScuApWk

Also IBM had a C++ compiler version for OS/2 based on their Visual Age for... stack that used an image store for code, offering a Smalltalk like editing experience for C++, but it was quite resource heavy for early 90's machines.

It had something to do with their CSet++ offering.

Re: Incremental Compilation

#36

Post by vidarh »

fulafel wrote: post #30
_RPM wrote: post #29

Since rust compiles to native machine code (bytes), how does it calculate the starting address of the code? for example, If there is a JMP instruction, JMP takes an address as an operand most likely -- Doesn't the kernel determine the starting address, or is the starting address the address returned by mmap?

Modern amd64 code is position-independent, so the JMP address is relative. On other architectures, it's the linker's job to convert symbolic addresses and it can choose all the addresses in one go as it's the final stage producting the executable.

That solves internal offsets. But code can also be loaded at different addresses on many platforms/architecture combos. That's solved by relocation tables: For every address, the linker will assume a starting address of 0, and then add the location of the address to a table. The runtime linker will then go through the relocation tables and fix up any entries with the real address.

Re: Incremental Compilation

#37

Post by paulasmuth »

pcwalton wrote: post #18
jjnoakes wrote: post #14

Earlier quoted context omitted.

Splitting up code into headers and sources also helps keep interfaces separate from implementations. It allows one to not only read just the header for the full interface, but it also allows one to distribute the implementation in binary form and the interface in source form, which is important for companies.

Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets. Not to mention that the size of your instance variables leaks into your public interface unless you manually heap allocate and use the pimpl idiom. In any case, Rust stores the interfaces to libraries in serialized binary form (including documentation, following a standardized form…

> Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets.

The C++ ABI is currently not portable anyway. So the concern about templates (or any C++ features) forcing you to put the implementation into the header file would not apply in the scenario dllthomas was referring to:

If you want to distribute your library as a binary object and a separate source-form interface/header today, you'll have to use a (wrapper) C API. Regardless of how "modern" the C++ code is.

Of course, it would be nice to have portable C++ objects some time in the future which would change things... :)

Re: Incremental Compilation

#38

Post by ngrilly »

mtanski wrote: post #16
bla2 wrote: post #7

From an uninformed distance this looks a bit underwhelming: initial builds are slower, incremental builds are not _that_ much faster, and build outputs are no longer deterministic functions of just the build inputs (which is useful for example for distributed build systems).

The way Rust works today is that you compile a whole crate together (all the files). The side effect of that is that the compiler can optimize the whole create vs what's in one file. You get an effect that analogous to LTO (like in C/C++). Conversely, if you now go to incremental builds the opportunity for LTO style optimizations. Thus you end up with a worse runtime performance in incremental compilation. Caveats an…

> Thus you end up with a worse runtime performance in incremental compilation.

Are you sure about this (in the current implementation)?

Re: Incremental Compilation

#39

Post by barrkel »

pcwalton wrote: post #6
skybrian wrote: post #3

Nice to see it. Incremental compilation is one of those things that production compilers have but research languages usually don't have. You only need it when you have a lot of code, but then you really need it.

Production compilers often don't have it either. (Having to split up your code manually into separate .cpp [or what have you] files doesn't qualify as incremental compilation—that's just separate compilation. Incremental compilation is when the compiler automatically figures out what needs to be recompiled, even when all you hand it is a big blob of code that hasn't been manually split up by a human in any way.)

The Delphi compiler has incremental compilation, but by your definition it wouldn't.

Delphi uses the source to figure out dependencies. That is, the compiler has make logic, and traces through the 'uses' declarations to recursively discover all the source and object files. It compares timestamps to discover which object files are out of date and need to be recompiled. Thus, if you touch a single file and do a compile, only a couple of files will be compiled - the main program source and the modified file. This dependency logic is in the compiler, not a separate make program.

If this isn't the compiler doing incremental compilation, I think we have a terminology problem. Because this is what is meant by incremental compilation as the term is applied to almost all production compilers, and if you persist in making a distinction, you may be confusing more than clarifying. ISTM that you're memoizing more parts of the compilation process than has historically been done in incremental compilers. It may be clearer to use a more qualified phrase to express this, rather than shift the generally accepted meaning of a term.

Re: Incremental Compilation

#40

Post by Raphael_Amiard »

Great blog post ! One thing I wonder as a compiler head though is, what is the granularity of this stuff ? Is it per-file or per-function, or even per block ? If it's finer than per file, how do you do the tree diff ? Do you parse whole files and then do a node-by-node diff ? Do you have an incremental parser ?

If I was to implement incremental compilation, I'd start with per-module I guess, because it's the atomic unit for a compiler, so it would be a lot simpler. That's why I'm curious.

Post reply on HN