Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ruby is too slow for programming competitions

blog.clifreeder.com

Post reply on HN
Print view View original

121–130 of 254 posts

Re: Ruby is too slow for programming competitions

#121

Post by chc »

bvdbijl wrote: post #36
nikic wrote: post #20

I also come to the conclusion that Ruby-like languages are just not the right tool for this kind of problem, but it should be pointed out that you would have been able to solve the 10^14 data set easily, even with Ruby. I was solving that problem with PHP and got it running reasonably fast (1s per range) for ranges up to around 10^60. PHP is probably faster than Ruby, but it's still in the same area. The first trick…

Yep, in a competition you want a language that had as little magic as possible. Languages like Ruby (not necessarily dynamic languages!) are just not really suitable for it

This seems like such a vague, insubstantial piece of advice. I've seen it before, but never in a more concrete form. I don't see any "magic" in Ruby. How do I determine if a language is magic? Usually, the best I can figure is that they're either confusing syntax with semantics or they're actually thinking of some library like ActiveRecord rather than Ruby.

Really, for pretty much anything I could conceivably think of to call "magic" in Ruby (e.g. method_missing), there's equivalent "magic" in Python (e.g. __get_attr__). And yet PyPy is pretty darned fast.

Re: Ruby is too slow for programming competitions

#122

Post by pak »

Derbasti wrote: post #84
RyanZAG wrote: post #77

A lot of people are talking about the problem itself - which is interesting - but very often you run into real problems which cannot be simplified so easily (NP hard problems, many different other cases). In addition, when you need to do important calculations on a server for your product and you're using Ruby, you may very well end up requiring 10x as many servers. This is pretty massive when you consider an AWS ser…

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

> it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

Agreed. Contrary to what lbrandy said, I've tried it, and it works very well. Writing Ruby extensions in C, or wrapping C libraries for Ruby, is really easy. Wrapping a PNG library for Ruby allowed us to draw about a trillion pixels' worth of images based on the human genome across a modest cluster of ~100 cores in a few days [1,2,3]. For reference, a terapixel image is about ~4TB of uncompressed data and ~800GB compressed, with Microsoft putting up comparable numbers for a project they ran on a fancier HPC cluster in 2010 [4]. We were heavily compressing it into PNG tiles on the fly so ours came down to ~60GB.

Regarding pointers to pointers to ints--if you keep all the data within the heap of the Ruby extension, as we did with the wrapped PNG library, then it can be stored in whatever simple data structure you want.

[1] http://chromozoom.org/?db=hg19

[2] https://github.com/rothlab/chromozoom

[3] http://bioinformatics.oxfordjournals.org/content/29/3/384.fu...

[4] http://research.microsoft.com/en-us/projects/terapixel/

Re: Ruby is too slow for programming competitions

#123

Post by jarrett »

Derbasti wrote: post #84
RyanZAG wrote: post #77

A lot of people are talking about the problem itself - which is interesting - but very often you run into real problems which cannot be simplified so easily (NP hard problems, many different other cases). In addition, when you need to do important calculations on a server for your product and you're using Ruby, you may very well end up requiring 10x as many servers. This is pretty massive when you consider an AWS ser…

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

Indeed. And there is more than one way to do it. The most obvious thing is to write a Ruby C extension. But you could also write a totally separate worker process in the language of your choice and glue it to your main Ruby app with a message queue. In most real-world cases, either solution will get you the performance you need to avoid 10xing your servers.

Re: Ruby is too slow for programming competitions

#125

Post by geedy »

davidw wrote: post #66

Great, so stop dicking around with some zero-sum game ( http://en.wikipedia.org/wiki/Zero%E2%80%93sum_game - a competition may only have one winner ) and go build a business, where no one cares what language you use as long as the solution works for your customers - and there can be more than one winner in many cases. Or even if you don't have a side project or startup or something, go build something cool and open s…

Curious, would you call University education dicking around as well?

Seems to me challenging the brain is never a waste of time. We would not have some of the medical technologies we have today if it were not for massively parallel GPUs, which of course were developed for the sole purpose of playing games.

Re: Ruby is too slow for programming competitions

#126

Post by PySlice »

nossralf wrote: post #90
trailfox wrote: post #65

Earlier quoted context omitted.

> Wouldn't you want to use a language that is familiar in a competition? If Ruby is your forte, surely you should use it? If you're familiar with knives why not bring a knife to a gun fight? If knives are your forte, surely you should use it?

If the goal of the competition is to slice apples, then a knife is pretty good compared to a gun. Or if the competition takes place under water. Or if you're Steven Seagal and the ship where you work is taken over by a splendidly over-acting Tommy Lee Jones and his gun-wielding minions. Execution speed is nice, I don't disagree. But it's not always the be-all and end-all deciding factor for what tool to use to solve…

> bring a knife to a gun fight

> If the goal of the competition is to slice apples

What?

Re: Ruby is too slow for programming competitions

#127

Post by lbrandy »

Derbasti wrote: post #84
RyanZAG wrote: post #77

A lot of people are talking about the problem itself - which is interesting - but very often you run into real problems which cannot be simplified so easily (NP hard problems, many different other cases). In addition, when you need to do important calculations on a server for your product and you're using Ruby, you may very well end up requiring 10x as many servers. This is pretty massive when you consider an AWS ser…

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

Everyone always agrees with this and, in my experience, has never really tried it. It works better in theory than in practice (or, at a minimum, it works only on a small subset of the types of problems that a naive view might otherwise lead you to believe). The problem is the data isn't organized in a way to be computation efficient, and all the C code in the world isn't going to make chasing pointers to pointers to pointers to ints fast.

A ton of work has to go into computation-friendly data organization (see all of numpy, scipy, etc) before you can ever actually optimize effectively.

EDIT: since all of my responses are of the form: "oh no I did it once with great results", I'll retract the part where I said "have never really tried it". I'm not arguing it's never worked. I'm arguing that the set of likely performance problems and the set of easy-to-optimize-with-C problems don't overlap as much as people would like to believe.

Re: Ruby is too slow for programming competitions

#128

Post by frehpt »

Derbasti wrote: post #84
RyanZAG wrote: post #77

A lot of people are talking about the problem itself - which is interesting - but very often you run into real problems which cannot be simplified so easily (NP hard problems, many different other cases). In addition, when you need to do important calculations on a server for your product and you're using Ruby, you may very well end up requiring 10x as many servers. This is pretty massive when you consider an AWS ser…

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

Option A:

  -Write your application in Go
Option B:

  -Write application in Ruby
  -Profile Ruby application to identify code to be rewritten in C
  -Learn C if you don't already know it
  -Rewrite various parts of your application in C
  -Run Valgrind to search for memleaks
  -Ensure that your target systems all have required libs installed
Option A seems much simpler...

Re: Ruby is too slow for programming competitions

#129

Post by pak »

danking00 wrote: post #116
Derbasti wrote: post #84

Earlier quoted context omitted.

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

Seconded. I recently wrote a image searching program (find a set of images as sub-images in another set of images) and I wrote most of it in Racket (Scheme dialect). I wrote the inner loop to calculate the correlation matrix in C and made an FFI call. I'm not familiar with Ruby's FFI, but I found making FFI calls dead simple in Racket.

FFI is also dead simple in Ruby.

https://github.com/ffi/ffi

It's pretty much `gem install ffi` and off you go. I've used a Ruby gem [1] that talks to a hashtable storage library via FFI and found it perfectly stable.

[1] https://github.com/jmettraux/rufus-tokyo

Re: Ruby is too slow for programming competitions

#130

Post by davidw »

geedy wrote: post #125
davidw wrote: post #66

Great, so stop dicking around with some zero-sum game ( http://en.wikipedia.org/wiki/Zero%E2%80%93sum_game - a competition may only have one winner ) and go build a business, where no one cares what language you use as long as the solution works for your customers - and there can be more than one winner in many cases. Or even if you don't have a side project or startup or something, go build something cool and open s…

Curious, would you call University education dicking around as well? Seems to me challenging the brain is never a waste of time. We would not have some of the medical technologies we have today if it were not for massively parallel GPUs, which of course were developed for the sole purpose of playing games.

> would you call University education dicking around as well?

That's a big debate! For a lot of people, I would, though, yes. There are serious economists who posit that universities are mostly about signalling, rather than actually learning much. I don't think I agree with that entirely, and believe it depends a lot on what's studied. It also depends on the opportunity costs, which may not be that high for many people that age.

> GPUs, which of course were developed for the sole purpose of playing games.

I don't believe the creation of games is a zero-sum game at all; there can be many winners, and, as you say, there are other side effects that are beneficial. The creation of games also falls into one of business, side-project or open source project as mentioned in my other post, no?

Artificial coding competitions aren't really about creating much of anything though, from what I can tell. A bunch of people solve the same thing and then throw away what they made.

I agree that challenging your brain isn't a waste, but it seems that in a world that is desperate for coders, there are more productive ways of challenging your brain in a way that's perhaps even more satisfying, because of the positive feeling of having helped to create something useful.

Post reply on HN