Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClojureC, a compiler for Clojure that targets C as a backend

github.com

Post reply on HN
Print view View original

81–89 of 89 posts

Re: ClojureC, a compiler for Clojure that targets C as a backend

#81

Post by bbq »

Rayne wrote: post #50
bbq wrote: post #40

Earlier quoted context omitted.

If the startup slowness is your problem, look into Nailgun: http://www.martiansoftware.com/nailgun/ If that's not your problem with the JVM, what is?

Nailgun is not a good solution to the problem. If the JVM startup speed is a problem, your best option is to use something not on the JVM and not use hacks that make it feel faster.

That's not true. Yes, it's worth considering outside of the JVM if being on the JVM means your application isn't interactive enough (startup speed doesn't matter for long running, fire & forget tasks).

At the same time, splitting your application into a client/server architecture is not a hack but an engineering decision. There are times when this decision is natural e.g. Music Player Daemon (MPD)[1]. For most CLI applications, there's no clear benefit (but the general approach has no clear downside either - the code overhead of this approach can be brought very low).

Certainly, in a production application you would want to secure the messaging channel (Nailgun doesn't).

[1] A music playing server: http://www.musicpd.org/. Some of the clients happen to be command line: http://mpd.wikia.com/wiki/Clients#Command_Line_Clients

Re: ClojureC, a compiler for Clojure that targets C as a backend

#82

Post by vidarh »

coldtea wrote: post #77
vidarh wrote: post #32

Earlier quoted context omitted.

This argument can just as easily be used to support a claim that generating C via string manipulation can very well be less error prone than relying on a huge, complex API like LLVM. It is not a given that there are "more moving parts" in generating C output from a compiler than in using LLVM.

> It is not a given that there are "more moving parts" in generating C output from a compiler than in using LLVM. No, but it's a given that the LLVM moving parts have been already written, and are tested by millions. Your moving parts in your own solution, you'd have to write yourself.

The LLVM moving-parts, yes, but you need to write code to use them too, and it's not a give that there are more moving parts in generating C output that those.

> Your moving parts in your own solution, you'd have to write yourself.

That's not always a bad thing for error rates, if the alternative is figuring out to use a massive library correctly.

Re: ClojureC, a compiler for Clojure that targets C as a backend

#83

Post by vidarh »

lobster_johnson wrote: post #69
vidarh wrote: post #38

Earlier quoted context omitted.

When you spend a few years speculating about what it would take to efficiently compile Ruby as statically as possible (I love Ruby, but I hate moving parts), devious becomes second nature... The idea of speculatively looking at method names comes from testing that to create vtables ahead of time for Ruby classes, to avoid hash tables in the common-case. As it turns out, most method on most Ruby classes are the ones i…

Out of interest, are you actually working on something like this for MRI? As you say, Ruby is in desperate need of optimization.

No, I've been off-and-on, toying with writing a "as static as possible" Ruby compiler (see http://www.hokstad.com/compiler) - it's been about two years since I last posted an update, but I have one new part complete and another one mostly complete. Just holding off posting for a bit longer because I want to have a bit of a buffer (3-4 complete parts) before I get peoples hope of regular posts up again...

What is there uses vtable's exclusively - I effective punted on the slow path (and so on adding methods at runtime) completely, but keep track of how much of the vtable allocations is wasted space. If/when I get there, the goal is to use various mechanisms like this to determine when to fall back on a slow path, and couple both with polymorphic inline caching when suitable.

EDIT:

I don't see MRI as very interesting to work on, largely because interpreters aren't much fun, and ironically given the amount of time I spend using Ruby, I prefer compilers to be as static as possible. I also prefer my compilers to be bootstrapped in their target language. Hence my "ideal" Ruby compiler would be written in pure Ruby, do a ton of static analysis, with minimal fallback to JIT when users user features that are too dynamic to analyse fully ahead of time

E.g. there's a ton of annoying uses of eval() in Ruby code where a more complete meta-programming API would make it trivial for a compiler to do full ahead of time static analysis, so one big thing an AOT Ruby compiler really need to do is to provide a library of compiler specific meta-programming facilities with a fallback that uses eval() as needed, and either convince people to use it, or provide monkey-patches for a number of popular projects. Some of these uses don't even need eval() in the first places, but uses it just as a quick shortcut because it's simpler...

Just to make clear, I'm not sure when or even if my compiler project will ever get to a state where it's even remotely useable to compile Ruby. I started it out without even having decided to compiler Ruby, mostly to write about various parts of the process of writing a compiler that I find interesting. I find compiling Ruby as incredibly fascinating from a theoretical point of view because of the complexity involved, but unfortunately working on it takes a lot more time and effort than thinking about the problems.

Re: ClojureC, a compiler for Clojure that targets C as a backend

#84

Post by vidarh »

coldtea wrote: post #75
rryan wrote: post #22

Earlier quoted context omitted.

Not everything can (or should) be proved of the drop of a hat in a discussion list If you say something that doesn't make sense, you're saying that one shouldn't have to back that up? It's like asking me to prove why using an XML processor to crete and save a DOM tree would produce more error free results than manually compiling tags as strings. It is in fact the equivalent of asking you to prove that an XML parser a…

>> Not everything can (or should) be proved of the drop of a hat in a discussion list >If you say something that doesn't make sense, you're saying that one shouldn't have to back that up? Read what you quoted from me, and what you ask. Where do I say that people should not back up things they say that "don't make sense"? Where do I even say they do not have to "back up" the things they say? I merely say that they do…

> I'm saying that a structured way to do that (LLVM API) is safer than merely creating strings yourself.

That is resting on the assumption that using the LLVM API is less error prone to a typical compiler developer than creating strings.

You are also assuming that spitting the C code to disk needs to be done in an unstructured way.

Neither of these assumptions are self evident.

Re: ClojureC, a compiler for Clojure that targets C as a backend

#85

Post by lobster_johnson »

vidarh wrote: post #83
lobster_johnson wrote: post #69

Earlier quoted context omitted.

Out of interest, are you actually working on something like this for MRI? As you say, Ruby is in desperate need of optimization.

No, I've been off-and-on, toying with writing a "as static as possible" Ruby compiler (see http://www.hokstad.com/compiler ) - it's been about two years since I last posted an update, but I have one new part complete and another one mostly complete. Just holding off posting for a bit longer because I want to have a bit of a buffer (3-4 complete parts) before I get peoples hope of regular posts up again... What is the…

I see. Sounds like an interesting project.

Sure, MRI is boring, but it's the best implementation right now (though some may argue that JRuby is better), and it's in desperate need of VM innovations.

Any new compiler/VM starting from scratch will be years away from being available for use in production environments. By the time it's finished, we'll all be using Go. (Sigh.)

Re: ClojureC, a compiler for Clojure that targets C as a backend

#86

Post by vidarh »

lobster_johnson wrote: post #85
vidarh wrote: post #83

Earlier quoted context omitted.

No, I've been off-and-on, toying with writing a "as static as possible" Ruby compiler (see http://www.hokstad.com/compiler ) - it's been about two years since I last posted an update, but I have one new part complete and another one mostly complete. Just holding off posting for a bit longer because I want to have a bit of a buffer (3-4 complete parts) before I get peoples hope of regular posts up again... What is the…

I see. Sounds like an interesting project. Sure, MRI is boring, but it's the best implementation right now (though some may argue that JRuby is better), and it's in desperate need of VM innovations. Any new compiler/VM starting from scratch will be years away from being available for use in production environments. By the time it's finished, we'll all be using Go. (Sigh.)

I think we need both. MRI can keep getting better, as can JRuby (which is an amazing feat, but to me, running on top of the JVM makes it a non-starter) or Rubinius, but they're fundamentally side-stepping the really hard problems.

E.g. nothing will stop MRI from having to interpret thousands of lines of code each time because it can't draw a line between runtime and compile time, while for an ahead of time compiler for Ruby, finding a pragmatic line between what needs to be executed at runtime vs. compile time is essential (consider for example the tendency to do stuff like getting the list of files in a directory and require all of them in turn).

Re: ClojureC, a compiler for Clojure that targets C as a backend

#87

Post by lobster_johnson »

vidarh wrote: post #86
lobster_johnson wrote: post #85

Earlier quoted context omitted.

I see. Sounds like an interesting project. Sure, MRI is boring, but it's the best implementation right now (though some may argue that JRuby is better), and it's in desperate need of VM innovations. Any new compiler/VM starting from scratch will be years away from being available for use in production environments. By the time it's finished, we'll all be using Go. (Sigh.)

I think we need both. MRI can keep getting better, as can JRuby (which is an amazing feat, but to me, running on top of the JVM makes it a non-starter) or Rubinius, but they're fundamentally side-stepping the really hard problems. E.g. nothing will stop MRI from having to interpret thousands of lines of code each time because it can't draw a line between runtime and compile time, while for an ahead of time compiler f…

We do need both. But some of the things you mention (like constructing vtables) could be applied to MRI's VM model without writing a compiler from scratch. Frankly, I would much rather have large performance improvements now than in 2-3 years.

(I agree about JRuby. I also wonder why Rubinius, which showed so much promise at the beginning, has stagnated. Is it simply the lack of developers?)

Re: ClojureC, a compiler for Clojure that targets C as a backend

#88

Post by vidarh »

lobster_johnson wrote: post #87
vidarh wrote: post #86

Earlier quoted context omitted.

I think we need both. MRI can keep getting better, as can JRuby (which is an amazing feat, but to me, running on top of the JVM makes it a non-starter) or Rubinius, but they're fundamentally side-stepping the really hard problems. E.g. nothing will stop MRI from having to interpret thousands of lines of code each time because it can't draw a line between runtime and compile time, while for an ahead of time compiler f…

We do need both. But some of the things you mention (like constructing vtables) could be applied to MRI's VM model without writing a compiler from scratch. Frankly, I would much rather have large performance improvements now than in 2-3 years. (I agree about JRuby. I also wonder why Rubinius, which showed so much promise at the beginning, has stagnated. Is it simply the lack of developers?)

I agree the performance increase would be great, but I think it needs to come gradually to MRI. E.g. trying to do anything fancy with the old AST-based interpreter would've been pretty pointless. After YARV, it is probably starting to get more attractive, but at the same time they've added method caching which gives a decent amount of the benefits. A vtable will still get faster, but it might not be the most immediately expedient way of speeding things up vs. e.g. Sasadas latest project of adding a generational gc.

Regarding Rubinius, writing compilers for dynamic languages is hard. Most textbooks you'll find cover techniques most suitable for statically typed languages (the best resource I know for starting to catch up on compiling dynamic languages is actually the Self papers). So you need more than an unusual level of interest in writing compilers to be likely to try to tackle a language like Ruby which is tricky even for dynamic languages (e.g. my favorite pet problem to meditate on: What constitutes 'compile time' vs. 'runtime' for ahead of time compiled Ruby?), and even more to actually persevere until you start getting proper results where you can get decent results in days with a simpler language.

It's made worse because of Ruby's horrendous grammar. And I mean that from a compiler writers perspective - as a developer I love to use Ruby to a large extent because the complexities of the grammar means it reads and writes better 95% of the time. But MRI's bison based parser was 6k-7k lines with ugly parser/lexer interplay last time I checked... There are full compilers substantially smaller than that for other languages...

To me, these complexities are part of what makes it fascinating. I firmly believe you can parse Ruby fully with a much, much simpler parser for example. A lot of the ugliness can be abstracted away, and C parser code is rarely good examples of succint code.

I did start playing with MRI years ago, specifically the parser, actually, and started chopping out redundant pieces, but got frustrated and bored with it. That's part of the problem - it's one thing to play around with a toy compiler like I've done, and another entirely to put in the effort to push a major change to MRI through to production quality given the number of years of accumulated history encapsulated in it. Doing the latter as a hobby is a daunting task.

Re: ClojureC, a compiler for Clojure that targets C as a backend

#89

Post by lobster_johnson »

vidarh wrote: post #88
lobster_johnson wrote: post #87

Earlier quoted context omitted.

We do need both. But some of the things you mention (like constructing vtables) could be applied to MRI's VM model without writing a compiler from scratch. Frankly, I would much rather have large performance improvements now than in 2-3 years. (I agree about JRuby. I also wonder why Rubinius, which showed so much promise at the beginning, has stagnated. Is it simply the lack of developers?)

I agree the performance increase would be great, but I think it needs to come gradually to MRI. E.g. trying to do anything fancy with the old AST-based interpreter would've been pretty pointless. After YARV, it is probably starting to get more attractive, but at the same time they've added method caching which gives a decent amount of the benefits. A vtable will still get faster, but it might not be the most immediat…

Just a note on Rubinius: The PyPy guys seem to have done pretty well at this. I don't know how similar they are to Rubinius; PyPy reduces to a RPython as an initial step, whereas I believe Rubinius compiles to LLVM's LI.
Post reply on HN