Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our experience using Clojure to speed up Beanstalk

blog.beanstalkapp.com

Post reply on HN
Print view View original

11–20 of 75 posts

Re: Our experience using Clojure to speed up Beanstalk

#11

Post by DanI-S »

dsabanin wrote: post #9
DanI-S wrote: post #7

I've been playing with 4clojure[1] in my off time for weeks, and it's been a great introduction to the language, although it obviously doesn't help with learning how to package and deploy an application or service. How did you make the first steps from playing around in the REPL to writing production code? [1] http://www.4clojure.com/

Have you seen Leiningen[1]? It pretty much solves packaging and deploying problem. Awesome tool. Also if you're into Web development, I can recommend playing with Noir[2] framework and Korma[3] for SQL abstraction. Heroku also support deploying Clojure apps out of the box, so you can easily use a free tier to get something out there. [1] https://github.com/technomancy/leiningen [2] http://webnoir.org/ [3] http://sqlk…

Thanks - I've actually played with Noir a little, but wasn't sure whether it was worth taking the time to explore fully. I'm glad to hear that it is!

Re: Our experience using Clojure to speed up Beanstalk

#12

Post by jwr »

michaelbuckbee wrote: post #8

I wonder to what extent the speed increase was due to Clojure vs the JVM itself. An interesting comparison might have been Clojure vs their previous code on JRuby.

In my code, there are two kinds of places where I get a performance boost by using Clojure:

1. Micro-optimizations, mostly due to JVM and its excellent JIT (the garbage collectors are quite impressive, too, if what you need is predictable response time).

2. Architectural gains: thanks to the Clojure's excellent concurrency support I can make much better use of multiple cores. I get more parallellism, hence better performance on same hardware.

The first kind is cool, because you get it "for free". The second kind is the real game-changer, because non-parallel software only gets you so far in terms of performance, and writing concurrent software is Hard. Clojure makes it much, much easier.

But overall I wouldn't say that Clojure is a performance daemon on a single CPU. You can get performance similar to carefully written Java code. This is good, but you can always do better with C or hand-written assembly on critical sections. But that's not the main advantage: the big thing is that I can write correct Clojure code fast, it runs well enough, and I can easily make use of multiple cores. You can debate micro-benchmarks all you want, but what really counts for me is how quickly (and correctly) I can get from zero to production code that runs fast enough.

Re: Our experience using Clojure to speed up Beanstalk

#13

Post by trimbo »

> I’ve been looking for an excuse to use Clojure in a production environment for a while

Except in wanting to mess around with Clojure, you didn't use the best tool for the job.

The Clojure solution now uses parallel implementations of Git and SVN to solve the problem, rather than the core code of SVN and Git. And now you also have a one-off daemon written in Clojure. It doesn't have the same support structure, ops requirements, or anything, as your Ruby code. Virtually no one uses clojure, so hiring and training are different, etc. You've incurred a lot of overhead for something not that great.

The best tool for this job was to improve the Ruby version by way of C extensions, or write a new C command that does this work for you, linking in the Git and SVN code directly. This has little to no new concepts, is straightforward, and would have given you the best compatibility and performance.

Re: Our experience using Clojure to speed up Beanstalk

#14

Post by dsabanin »

dons wrote: post #10

http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Median speedup for the same problem in Clojure was 12x over Ruby. and 14x for Haskell; ... and 25x in Java .. and 35x in C++...

Unfortunately the cost of developing this solution in C++ would be forbidding for us.

In our case on the pretty much same loop the difference between Ruby and Clojure was 20-30 times. Please note that using Clojure allowed us direct access to Java libraries, so essentially this is Java performance that we've been able to tap into using Clojure.

Re: Our experience using Clojure to speed up Beanstalk

#15

Post by dsabanin »

trimbo wrote: post #13

> I’ve been looking for an excuse to use Clojure in a production environment for a while Except in wanting to mess around with Clojure, you didn't use the best tool for the job. The Clojure solution now uses parallel implementations of Git and SVN to solve the problem, rather than the core code of SVN and Git. And now you also have a one-off daemon written in Clojure. It doesn't have the same support structure, ops r…

Clojure code is 700 lines, and pretty easy to grasp by anyone who've used Lisp before. We would have to implement a lot of C code to speed up both Git and Svn bindings. And I seriously doubt that we would be able to debug this in a reasonable amount of time.

Clojure caching took me like a month max.

Re: Our experience using Clojure to speed up Beanstalk

#16

Post by Patient0 »

"The rewrite in Clojure resulted in much cleaner, faster code, totaling at only 700 lines of Clojure code (I don’t have a clear comparison with Ruby code here). "

Could you post the Clojure code somewhere? It would be very interesting to see clean, fast Clojure code written to solve a real-world problem (as opposed to some toy example).

Re: Our experience using Clojure to speed up Beanstalk

#17

Post by trimbo »

dsabanin wrote: post #15
trimbo wrote: post #13

> I’ve been looking for an excuse to use Clojure in a production environment for a while Except in wanting to mess around with Clojure, you didn't use the best tool for the job. The Clojure solution now uses parallel implementations of Git and SVN to solve the problem, rather than the core code of SVN and Git. And now you also have a one-off daemon written in Clojure. It doesn't have the same support structure, ops r…

Clojure code is 700 lines, and pretty easy to grasp by anyone who've used Lisp before. We would have to implement a lot of C code to speed up both Git and Svn bindings. And I seriously doubt that we would be able to debug this in a reasonable amount of time. Clojure caching took me like a month max.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here).

Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker news. Clojure is not it for all the reasons I mentioned. And if people think C is hard, practice it. Read Zed's book and do Project Euler problems with it until you feel comfortable with it. It's an essential tool everyone must know how to reach for.

Re: Our experience using Clojure to speed up Beanstalk

#19

Post by swannodette »

dons wrote: post #10

http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Median speedup for the same problem in Clojure was 12x over Ruby. and 14x for Haskell; ... and 25x in Java .. and 35x in C++...

Alioth benchmarks are out of date as far as Clojure is concerned. These days it's not much work to get Clojure to deliver identical to Java performance. Better versions of the same benchmarks here that show the same performance as Java that you verify yourself - http://github.com/clojure/test.benchmark

Re: Our experience using Clojure to speed up Beanstalk

#20

Post by spitfire »

trimbo wrote: post #17
dsabanin wrote: post #15

Earlier quoted context omitted.

Clojure code is 700 lines, and pretty easy to grasp by anyone who've used Lisp before. We would have to implement a lot of C code to speed up both Git and Svn bindings. And I seriously doubt that we would be able to debug this in a reasonable amount of time. Clojure caching took me like a month max.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

They're optimizing for their business value and resources (programmers) available.

It might not impress their programmer friends, but it'll impress their accountant.

Post reply on HN