Live data from Hacker News

How Microsoft rewrote its C# compiler in C# and made it open source (2017)

medium.com

81–90 of 136 posts

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#81
post #50

While the rewrite enabled faster development of the language as a whole, it also gradually destroyed the IDE's performance. The editor in VS 2019 is simply unworkable. I blame this directly on the immutable AST. While a nice concept in theory, it causes too many allocations, and is cumbersome to work with. I predict another rewrite in 2 or 3 years.

I found that VS 2019 is faster than VS 2017. But VS 2019 with Resharper is a lot more slower than VS 2017 with resharper. I would blame resharper there, not VS. VS never felt that fast than right now.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#82
post #50

While the rewrite enabled faster development of the language as a whole, it also gradually destroyed the IDE's performance. The editor in VS 2019 is simply unworkable. I blame this directly on the immutable AST. While a nice concept in theory, it causes too many allocations, and is cumbersome to work with. I predict another rewrite in 2 or 3 years.

If an immutable datastructure causes to many allocations surely that's an issue with the allocator rather than immutability?

I'm not familiar with this compiler but I'm currently writing a mostly immutable structure so I'm curious as to whether it's an issue.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#83
post #64

Roslyn's parser and syntax tree is pretty amazing. You can recreate the precise source text from the parse tree up to whitespace. This sort of "bijective parsing" is truly incredible and probably one of the cooler innovations I've seen in parsing technology. I can see a bunch of really interesting ideas that you could do with bijective parsing. For instance, imagine Rails style boilerplate generation but done at a se…

Pardon me if I'm missing something, but doesn't just about any AST implementation allow recovering the source text up to whitespace – or indeed including whitespace, given that most real-world parsers have to retain lexical information in order to support reasonable diagnostic messages anyway.

Many times things like comments are dropped and names are disambiguated. Printing the tree back out would yield an equivalent program but the source code would look way different.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#84
post #49

If .NET/C# could output native binaries for Windows Desktop apps, I could seriously think about switching to C# over C++. How do people deal with securing their source code otherwise? Quite trivial to get the source from a decompiled C# exe file.

It's quite trivial to get the source from a C# exe file in the same way as a decompiler makes it trivial to get the source from a C++ exe file. Obfuscators are common if you're concerned about the symbols leaking. If you don't include the .PDB debugging symbol files it's much the same as native binary code. The .NET virtual machine code is a little more expressive but is superficially similar to x86 native code. In m…

A C# binary can be decompiled back to easy to read source code. Show me how this can be done with a compiled C++ binary. It is a valid concern.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#85
post #8

I find amusing how microsoft moved from the closed source referent in the industry into such an open source player. Right now even allows to hook some of their tools and platforms to its competing platforms and tools.

Isn't this because they've jumped on the ad model perfected by google, facebook, etc? Open source it for press/good will, give it away for free and collect data.

"The world’s most valuable resource is no longer oil, but data"

https://news.ycombinator.com/item?id=14269073

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#86
post #80
post #64

Earlier quoted context omitted.

Pardon me if I'm missing something, but doesn't just about any AST implementation allow recovering the source text up to whitespace – or indeed including whitespace, given that most real-world parsers have to retain lexical information in order to support reasonable diagnostic messages anyway.

There is reason AST is called abstract syntax tree not just syntax tree. Many syntax details like parentheses or choice between alternative syntax forms are there only to avoid syntactic ambiguity, improve readability or historical reasons and don't affect the semantics or even error messages. It's not surprising for compiler authors to choose discarding some of the unnecessary information as early as possible to sav…

Notable a lot of JavaScript AST libraries at least a while back omitted comments and whitespace, which is fine for writing a minifier, but horrible when trying to implement source code transformations.

What I really like about Roslyn's AST (and I guess TypeScript's looks about the same) is that all those details are put into Trivia; ignoring those is just passing one flag to the visitor; you don't have to be aware of how many different types of AST nodes there are that might be ignorable. It also means that each and every whitespace and comment belongs to a syntax token instead of being other nodes interleaved with the normal nodes. This helps keeping comments where they belong even after re-arranging code in the AST – something that, e.g. ReSharper is atrociously bad at.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#87

I still can't believe more people aren't leveraging the Roslyn APIs to write compiler extensions or additional tools around C#. It's conceptually powerful.

We've written a custom compiler from C# to Java and JavaScript based on Roslyn. Over time it gained more features and target languages as well (Python is in progress, we can also emit a working GWT wrapper for the JavaScript output, we can emit TypeScript typings or just normal TypeScript as well, etc.). For us this helps us in offering our products on various different platforms without having to write the code in different places anew. Since our product is a library, not an application, we couldn't really take advantage of existing conversion tools that mostly take the path of converting IL to hideous code and an entry point.

The whole thing is now used in basically every library build we have at some point, even for the C# versions, as it ties in with our documentation writing process and places the correct API names and links for that product into the documentation, even though the docs start with mostly the same content for each.

I agree that lack of documentation makes working with Roslyn a bit daunting at times, although the API is very well designed and oftentimes it's very obvious where to look for something. I was also very impressed by their compatibility efforts. We started while Roslyn was in beta and upgrading through the releases worked without a hitch.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#88
post #50

While the rewrite enabled faster development of the language as a whole, it also gradually destroyed the IDE's performance. The editor in VS 2019 is simply unworkable. I blame this directly on the immutable AST. While a nice concept in theory, it causes too many allocations, and is cumbersome to work with. I predict another rewrite in 2 or 3 years.

I had to ditch ReSharper to get to 2019 because together with the performance of the IDE itself, it just wasn't usable. R# ate gigs of Ram and Roslyn does the same. It's not surprising since they basically do the same thing, in managed code! But I can't pay the CPU time and memory to analyze my code TWICE on every edit. I also suspect things like switching build configuration offers thousands of opportunities to have…

I highly suggest giving Rider (also by JetBrains) a try. They run most things in separate processes/threads instead of running everything in the UI thread like VS does. It's almost a drop-in replacement.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#89
post #80
post #64

Earlier quoted context omitted.

Pardon me if I'm missing something, but doesn't just about any AST implementation allow recovering the source text up to whitespace – or indeed including whitespace, given that most real-world parsers have to retain lexical information in order to support reasonable diagnostic messages anyway.

There is reason AST is called abstract syntax tree not just syntax tree. Many syntax details like parentheses or choice between alternative syntax forms are there only to avoid syntactic ambiguity, improve readability or historical reasons and don't affect the semantics or even error messages. It's not surprising for compiler authors to choose discarding some of the unnecessary information as early as possible to sav…

It's called abstract syntax tree because the concrete syntax is not a tree, it's an list of characters. Not because it makes semantic-invariant transformations.

Re: How Microsoft rewrote its C# compiler in C# and made it open source (2017)

#90
post #38
post #27

Earlier quoted context omitted.

Their intent is still entirely market share. We need to be vigilant on how they get there. History has taught us a lot of lessons we seem to have forgotten because shiny and new layer of marketing. There is still a massive cultural and technical impedance mismatch.

I agree. A lot of what they are doing now looks like EEE when you peel back a couple of layers. I say this as someone who happily uses a lot of the stuff they produce in the process. I really, really, intensely hope I'm wrong and my fears aren't realised.

If it's open source, how does extinguish work?
Post reply on HN