Observations from a real-world Clojure project • Hacker News Board

Re: Observations from a real-world Clojure project

#11

Post by gruseom » Thu, Oct 29, 2009, 11:23 PM UTC

smanek wrote: post #6

aidenn0 wrote: post #3

I'd like to reiterate the lack of good debuggers for not just Clojure, but also free CL implementations. I haven't branched out much recently, but is sbcl/SLIME still state-of-the-art for lisp debugging in the open source world?

I personally use SBCL, but I've heard a lot of people use Clozure Common Lisp ( http://www.clozure.com/clozurecl.html ) because it has better Slime integration (better stack frame inspector, etc). The very best tools are probably Allegro CL (w/ its non-emacs IDE) and Lisp Works (and its own IDE too). I've heard nothing but good things about both (and have used Allegro CL enough to know it - and its associated librari…

I've heard a lot of people use Clozure Common Lisp (http://www.clozure.com/clozurecl.html) because it has better Slime integration (better stack frame inspector, etc).

Really? I would have said CCL has worse Slime integration in general (Slime being an epiphenomenon of the SBCL-on-Linux world) and far worse when it comes to debugging. Am I missing something?

We're using CCL with Slime on OS X. The Clozure people are working on a new IDE, but I don't know how good its debugging facilities are yet. It would take a lot to get me off Emacs, but slick debugging might do it.

Alternatively, I often think that maybe we should just take the time to make sldb do what we want. But it's hard to take precious resources away from one's main project.

If anyone feels the same way and is interested in hacking on this, email me - maybe we can work something out.

Re: Observations from a real-world Clojure project

#12

Post by smanek » Thu, Oct 29, 2009, 11:38 PM UTC

gruseom wrote: post #11

smanek wrote: post #6

Earlier quoted context omitted.

I personally use SBCL, but I've heard a lot of people use Clozure Common Lisp ( http://www.clozure.com/clozurecl.html ) because it has better Slime integration (better stack frame inspector, etc). The very best tools are probably Allegro CL (w/ its non-emacs IDE) and Lisp Works (and its own IDE too). I've heard nothing but good things about both (and have used Allegro CL enough to know it - and its associated librari…

I've heard a lot of people use Clozure Common Lisp ( http://www.clozure.com/clozurecl.html ) because it has better Slime integration (better stack frame inspector, etc). Really? I would have said CCL has worse Slime integration in general (Slime being an epiphenomenon of the SBCL-on-Linux world) and far worse when it comes to debugging. Am I missing something? We're using CCL with Slime on OS X. The Clozure people ar…

IIRC, ITA Software (one of the biggest Lisp houses in the world ...) uses SBCL for their main QPX project. A few years ago when they were starting the new Polaris project, they decided to go with CCL instead of continuing with SBCL.

My understanding is that the two major motivating factors for this decision were that CCL compiles faster and has better debugging support.

This is all based on my memory of conversations that took place about a year ago, so it's entirely possible I'm mis-remembering or what used to be true no longer holds. I have no first hand experience with CCL so if you do, I'll take your word for it ...

Re: Observations from a real-world Clojure project

#13

Post by barrkel » Fri, Oct 30, 2009, 12:05 AM UTC

mcav wrote: post #8

nearestneighbor wrote: post #7

> Example: ~10M records are processed and transformed, various computations occur, and ~100K records are spit out. Lots of statistics. One type of run takes 12 hours on an 8GB, 4-core Linux box That seems insanely slow, unless they are doing something insanely complex. In other words, this doesn't tell us very much.

12 hours / 10,000,000 records = 4.32ms/record concurrently. (4 cores maxed out ≅ 16ms per item?) Doesn't tell us much, but that doesn't seem insanely slow either, necessarily.

Assuming pretty cache-optimal code and 1.5GHz ops with no superscalar gains, that's over 25 million operations per record.

Most PC video games, for example, expect a refresh rate somewhere between 30Hz and 60Hz. 60Hz gives you 17ms for an entire frame, in which it's doing an incredible amount of work: drawing a whole scene, updating the world, running physics, etc. etc.

I think you have too low expectations. Modern hardware is very, very fast.

Re: Observations from a real-world Clojure project

#14

Post by WalkingDead » Fri, Oct 30, 2009, 12:11 AM UTC

5. The functionality of the docs hasn't kept up with Clojure.

6. Debugging facilities also have not kept up with the state of Clojure.

These two are the first thing I check before trying to use any programming language.

Re: Observations from a real-world Clojure project

#15

Post by lucifer » Fri, Oct 30, 2009, 12:30 AM UTC

dgreensp wrote: post #9

nearestneighbor wrote: post #7

> Example: ~10M records are processed and transformed, various computations occur, and ~100K records are spit out. Lots of statistics. One type of run takes 12 hours on an 8GB, 4-core Linux box That seems insanely slow, unless they are doing something insanely complex. In other words, this doesn't tell us very much.

My "slow sense" is tingling too. I'm deadly curious to know how fast I could make that "12-hour" job using Scala or Java. I've done extensive work with Scala, Java, and bytecodes; writing performant Scala code requires some knowledge of the JVM and how Scala is compiled, and requires sacrificing some of its power; but because the static typing and method dispatching is so similar to Java's, you can get there. From ev…

"From everything I've read about Clojure, it sounds a whole level less efficient."

Curious as to what you have read regarding clojure performance. I recall hearing Cliff Click giving it a thumbs up at a JVM language summit:

http://www.infoq.com/presentations/click-fast-bytecodes-funn...

This may also be apropos: http://groups.google.com/group/clojure/msg/cccf532ca04fcdf4?...

Re: Observations from a real-world Clojure project

#16

Post by greendestiny » Fri, Oct 30, 2009, 12:41 AM UTC

barrkel wrote: post #13

mcav wrote: post #8

Earlier quoted context omitted.

12 hours / 10,000,000 records = 4.32ms/record concurrently. (4 cores maxed out ≅ 16ms per item?) Doesn't tell us much, but that doesn't seem insanely slow either, necessarily.

Assuming pretty cache-optimal code and 1.5GHz ops with no superscalar gains, that's over 25 million operations per record. Most PC video games, for example, expect a refresh rate somewhere between 30Hz and 60Hz. 60Hz gives you 17ms for an entire frame, in which it's doing an incredible amount of work: drawing a whole scene, updating the world, running physics, etc. etc. I think you have too low expectations. Modern h…

This whole discussion is silly, there isn't nearly enough information given in the post to infer anything. The author is just making the point that it isn't a toy application.

Re: Observations from a real-world Clojure project

#17

Post by yangyang » Fri, Oct 30, 2009, 12:48 AM UTC

wmf wrote: post #10

cma wrote: post #5

Earlier quoted context omitted.

You shouldn't be able to SEGV a JVM no matter what bytecode you feed it (exempting JNI). So it is a JVM bug.

OTOH it doesn't really matter whose "fault" it is; if using Clojure causes SEGVs but some other Lisp doesn't cause SEGVs, that's a problem.

That depends if you're using it primarily because it is a lisp or primarily because of its Java interop / the fact it runs on the JVM.

Re: Observations from a real-world Clojure project

#18

Post by jjs » Fri, Oct 30, 2009, 12:56 AM UTC

smanek wrote: post #12

gruseom wrote: post #11

Earlier quoted context omitted.

I've heard a lot of people use Clozure Common Lisp ( http://www.clozure.com/clozurecl.html ) because it has better Slime integration (better stack frame inspector, etc). Really? I would have said CCL has worse Slime integration in general (Slime being an epiphenomenon of the SBCL-on-Linux world) and far worse when it comes to debugging. Am I missing something? We're using CCL with Slime on OS X. The Clozure people ar…

IIRC, ITA Software (one of the biggest Lisp houses in the world ...) uses SBCL for their main QPX project. A few years ago when they were starting the new Polaris project, they decided to go with CCL instead of continuing with SBCL. My understanding is that the two major motivating factors for this decision were that CCL compiles faster and has better debugging support. This is all based on my memory of conversations…

Remember also that during the time that ITA would have made that decision, SBCL and SLIME were both evolving rapidly.

Re: Observations from a real-world Clojure project

#19

Post by tensor » Fri, Oct 30, 2009, 1:14 AM UTC

dgreensp wrote: post #9

nearestneighbor wrote: post #7

> Example: ~10M records are processed and transformed, various computations occur, and ~100K records are spit out. Lots of statistics. One type of run takes 12 hours on an 8GB, 4-core Linux box That seems insanely slow, unless they are doing something insanely complex. In other words, this doesn't tell us very much.

My "slow sense" is tingling too. I'm deadly curious to know how fast I could make that "12-hour" job using Scala or Java. I've done extensive work with Scala, Java, and bytecodes; writing performant Scala code requires some knowledge of the JVM and how Scala is compiled, and requires sacrificing some of its power; but because the static typing and method dispatching is so similar to Java's, you can get there. From ev…

How can anyone even approach this with a straight face? You don't even know what they are computing!

If you want to question clojure's performance for technical reasons, do that. But let's not use feelings, and instincts in discussions about concrete scientific topics. There have certainly been several interesting articles on clojure performance, and work done to improve performance for native data structures (transients). I, and no doubt many others, would be genuinely interested in performance studies on more complex concurrency problems I've seen so far.

Re: Observations from a real-world Clojure project

#20

Post by lucifer » Fri, Oct 30, 2009, 1:28 AM UTC

tensor wrote: post #19

dgreensp wrote: post #9

Earlier quoted context omitted.

My "slow sense" is tingling too. I'm deadly curious to know how fast I could make that "12-hour" job using Scala or Java. I've done extensive work with Scala, Java, and bytecodes; writing performant Scala code requires some knowledge of the JVM and how Scala is compiled, and requires sacrificing some of its power; but because the static typing and method dispatching is so similar to Java's, you can get there. From ev…

How can anyone even approach this with a straight face? You don't even know what they are computing! If you want to question clojure's performance for technical reasons, do that. But let's not use feelings, and instincts in discussions about concrete scientific topics. There have certainly been several interesting articles on clojure performance, and work done to improve performance for native data structures (transi…

Then this discussion between Cliff Click and Rich Hickey may be of interest: http://blogs.azulsystems.com/cliff/2008/05/clojure-stms-vs.h...

Its a bit dated ('08), but sounds like the jury is still out on STMs.