Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our experience using Clojure to speed up Beanstalk

blog.beanstalkapp.com

Post reply on HN
Print view View original

21–30 of 75 posts

Re: Our experience using Clojure to speed up Beanstalk

#21

Post by paulja »

trimbo wrote: post #17
dsabanin wrote: post #15

Earlier quoted context omitted.

Clojure code is 700 lines, and pretty easy to grasp by anyone who've used Lisp before. We would have to implement a lot of C code to speed up both Git and Svn bindings. And I seriously doubt that we would be able to debug this in a reasonable amount of time. Clojure caching took me like a month max.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

Before rails no-one used Ruby, the parallel implementations of svn and git you mention are heavily used in Eclipse and throughout the Java world. They gain access to all the advantages of the JVM. By using clojure they probably attract a better standard of developer. Why bother introducing a buggy blob of C when they can do it this way.

Re: Our experience using Clojure to speed up Beanstalk

#22

Post by shanemhansen »

beanstalkapp guys/girls, thanks for sharing! I'd be really interested to see some profiling results, were you bottlenecked on sha hashing? IO/execve syscalls? Memory usage?

I think the reason for the performance difference is pretty clear. According to the article and the grit documentation, all the git api calls were either done in pure ruby, or by shelling out to `git`. Also, reading between the lines, it sounds like they required some information/relationships on commits that was non-trivial to retrieve using a basic git shell command. So by switching out the runtime and the algorithm, they get a huge performance increase.

With jgit, they can more easily traverse the graph directly and efficiently. I'm really curious to see what kind of performance they could get from using FFI+jruby and raw c calls to libgit/libgit2 (http://libgit2.github.com/).

Re: Our experience using Clojure to speed up Beanstalk

#23

Post by trimbo »

spitfire wrote: post #20
trimbo wrote: post #17

Earlier quoted context omitted.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

They're optimizing for their business value and resources (programmers) available. It might not impress their programmer friends, but it'll impress their accountant.

> It might not impress their programmer friends, but it'll impress their accountant

If impressing accountants is the goal of what we do, then the whole thing should have been outsourced overseas.

Re: Our experience using Clojure to speed up Beanstalk

#24

Post by edwinnathaniel »

Patient0 wrote: post #16

"The rewrite in Clojure resulted in much cleaner, faster code, totaling at only 700 lines of Clojure code (I don’t have a clear comparison with Ruby code here). " Could you post the Clojure code somewhere? It would be very interesting to see clean, fast Clojure code written to solve a real-world problem (as opposed to some toy example).

Yes, please. I'd like to see "clean and fast Clojure code".

Re: Our experience using Clojure to speed up Beanstalk

#25

Post by [deleted] »

paulja wrote: post #21
trimbo wrote: post #17

Earlier quoted context omitted.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

Before rails no-one used Ruby, the parallel implementations of svn and git you mention are heavily used in Eclipse and throughout the Java world. They gain access to all the advantages of the JVM. By using clojure they probably attract a better standard of developer. Why bother introducing a buggy blob of C when they can do it this way.

[deleted]

Re: Our experience using Clojure to speed up Beanstalk

#26

Post by mrgoldenbrown »

trimbo wrote: post #17
dsabanin wrote: post #15

Earlier quoted context omitted.

Clojure code is 700 lines, and pretty easy to grasp by anyone who've used Lisp before. We would have to implement a lot of C code to speed up both Git and Svn bindings. And I seriously doubt that we would be able to debug this in a reasonable amount of time. Clojure caching took me like a month max.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

I think you are doing students and young programmers a disservice by implying that there is one "best" solution here, and that you know it. I am assuming you haven't seen the ruby or Clojure code involved, don't work at Beanstalk, don't know the skillset of their team, their budget, strategic plans, or any other one of a bunch of details that determine what is best in a particular situation. If you already know C then perhaps it would be right for you. That does not mean it is right for everyone.

Re: Our experience using Clojure to speed up Beanstalk

#27

Post by jhickner »

trimbo wrote: post #17
dsabanin wrote: post #15

Earlier quoted context omitted.

Clojure code is 700 lines, and pretty easy to grasp by anyone who've used Lisp before. We would have to implement a lot of C code to speed up both Git and Svn bindings. And I seriously doubt that we would be able to debug this in a reasonable amount of time. Clojure caching took me like a month max.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

Those same students and programmers should realize that yours is just one opinion among many. Your advice leans toward the "fungible cogs" school, where the capital crime is to expect others to learn something they don't already know.

That's still a widely held view, and maybe even a majority, but probably not on HN. (see pg's Beating the Averages essay: http://www.paulgraham.com/avg.html)

Re: Our experience using Clojure to speed up Beanstalk

#28

Post by trimbo »

mrgoldenbrown wrote: post #26
trimbo wrote: post #17

Earlier quoted context omitted.

Lines of code is the new form of premature optimization. A C solution is better in performance and compatibility, but you've prematurely optimized on lines of code as your deciding factor (based on you mentioning it in the article, and again here). Obviously, you can choose whatever you like, it's your company. I'm just explaining what the best solution was for the many students and young programmers who visit hacker…

I think you are doing students and young programmers a disservice by implying that there is one "best" solution here, and that you know it. I am assuming you haven't seen the ruby or Clojure code involved, don't work at Beanstalk, don't know the skillset of their team, their budget, strategic plans, or any other one of a bunch of details that determine what is best in a particular situation. If you already know C the…

> I think you are doing students and young programmers a disservice by implying that there is one "best" solution here, and that you know it.

I read their requirements in the blog post. They need fast access to SVN and Git internals to store off metadata. Based on that it's really not hard to determine what to use to solve this problem. There may not be one exact right answer (in fact, I suggested two), but there is a direction that makes sense and one that does not.

The real disservice to those coming up in this industry is reading blog post after blog post on Hacker News with people looking for "excuses" to use random languages they're interested in playing with. The answer to most of these problems is to use something really boring and straightforward, like C, C++ or Java. I guess I've worked too many places where legacy code becomes a huge burden because one guy wants to play around with X one time. The funny thing is, I see these same guys go onto the next place and do it there.

Re: Our experience using Clojure to speed up Beanstalk

#29

Post by cmelbye »

dsabanin wrote: post #9
DanI-S wrote: post #7

I've been playing with 4clojure[1] in my off time for weeks, and it's been a great introduction to the language, although it obviously doesn't help with learning how to package and deploy an application or service. How did you make the first steps from playing around in the REPL to writing production code? [1] http://www.4clojure.com/

Have you seen Leiningen[1]? It pretty much solves packaging and deploying problem. Awesome tool. Also if you're into Web development, I can recommend playing with Noir[2] framework and Korma[3] for SQL abstraction. Heroku also support deploying Clojure apps out of the box, so you can easily use a free tier to get something out there. [1] https://github.com/technomancy/leiningen [2] http://webnoir.org/ [3] http://sqlk…

I've got a question about Noir. Is it necessary for me to write HTML with Clojure, or can I separate it and write plain HTML somehow? Writing HTML in the middle of my Clojure code doesn't appeal to me coming from a MVC background...

Re: Our experience using Clojure to speed up Beanstalk

#30

Post by Arare »

swannodette wrote: post #19
dons wrote: post #10

http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Median speedup for the same problem in Clojure was 12x over Ruby. and 14x for Haskell; ... and 25x in Java .. and 35x in C++...

Alioth benchmarks are out of date as far as Clojure is concerned. These days it's not much work to get Clojure to deliver identical to Java performance. Better versions of the same benchmarks here that show the same performance as Java that you verify yourself - http://github.com/clojure/test.benchmark

"The Computer Language Benchmarks Game" has never been a valid benchmark for any language or implementation.
Post reply on HN