> i worry that caching pre-processed files is a red herring - its it really faster than re-including?
For future reference: In large C++ projects, it's not at all unusual for greater than 90% of the compile time to be spent parsing (and re-parsing, and re-parsing, and re-parsing, ad nauseam) header files.
> what about macros in include files?
The AST of the header is persisted. If the parser can parse macro definitions, macros will continue to work normally. The only case where you would need to fall back to #include is those rare times in which you want defining something in the source file to alter the parsing of the header (and even in most of those, you should just define the constant in the call to the compiler eg) "clang foo.c -DWITH_FEATURE_X")
> i feel that the preprocessor ultimately ends up with the same amount of work, just an extra pass for each included header to build a version to be cached… not to mention the complexity required to handle the multiplicity of pre-processor states required for this. maybe i am being dim and missing the obvious.
The pre-processor merely slurps the text of the included file into the including file; the compiler then parses the entire gigantic soup of + . The semantics of this require that every included header be re-parsed once per compilation unit. Imagine you have 5 .cpp files, each containing 200 characters, and each #including iostreams (which weighs in at roughly 1 million characters). Each .cpp file, post-preprocessor phase, will be 1 million, 200 characters long. A full compilation of the project will require the parsing of 5 million, 1 thousand characters. Any subsequent full build will require parsing the full 5 million, 1 thousand characters. Changing one .cpp file will result in the need to parse 1 million, 200 characters.
In this proposal, by contrast, an included file need only ever be parsed once; its AST can then be persisted and referenced eternally. In our above example, the iostreams header will be parsed once, and each .cpp file will be parsed once. This means a full build, the very first time iostreams is ever referenced in any compilation on the system, will require parsing 1 million, 1 thousand characters. Any subsequent full build will require parsing merely 1 thousand characters. Changing one .cpp file will require parsing merely 200 characters.