Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rails 3 Performance - Not Good Enough

blog.tstmedia.com

Post reply on HN
Print view View original

111–120 of 142 posts

Re: Rails 3 Performance - Not Good Enough

#111

Post by stiff »

aneth wrote: post #53

I've been a Rails dev for 5 years. I'm frequently considering leaving for another framework because of one thing: bootstrap time. Starting a test or server on my dev machine takes 20-30 seconds. Particularly with tests, this is a huge problem for doing proper TDD, particularly when you are trying to use tests to track down a bug. In that 20-30 seconds, I often console myself that we no longer need to print punch card…

I have investigated this in detail and the reason for this is really ridiciulous - 80% of this time is spend on executing "require" statements and this is due to really bad design of RubyGems and/or Bundler. They both do their job by augmenting $LOAD_PATH to include all directories containing gem contents. If you then look into how Ruby deals with the $LOAD_PATH, it turns out each time you do a "require" it will go through all of those directories and their subdirectories in search for a file matching what you have required. I used strace to see how this impacts doing "rails console" on an application with around 30 gems - it ended up doing 35000 open calls that ended up with ENOENT. It is beyond my mind how this can remain unfixed for so long. If RubyGems instead maintained a cache from all the gem directories, it could map requires to files without touching the filesystem and it would work many times as fast. Even creating a cache at runtime and then using it for the lookup would be many times as fast. Unfortunately, it is hard to determine a strategy for building this cache that would map the requires to files in exactly the same was as RubyGems, because RubyGems has some pretty weird strategy for determining which files take priority over which files when you do an ambiguous require. Because of this, I haven't yet succeeded in implementing a fix myself and I'm not 100% sure if it is possible. I also tried to contact one of the Bundler guys, but so far haven't had any reply about this.

Re: Rails 3 Performance - Not Good Enough

#112

Post by strmpnk »

cmelbye wrote: post #80
strmpnk wrote: post #58

Interesting, though I no longer use ActiveRecord in my largest app because of how slow it is (even in Rails 2). Try checking the overhead of instantiating (no save or db lookup) a simple model instance with 2 fields for example. If you manage to avoid that, you've beaten the slowest parts. The view engine is also slow but I tend to do rendering in the browser these days so that barely matters too me. If you are caref…

What would you suggest to use instead of ActiveRecord in a Rails 3 app?

It depends on the data. For one app I've written my own CouchDB driver which, as slow as CouchDB is known to be, seems to kill most AR equivalent code. That's pending release as soon as it's stable.

Generally speaking, I think AR's problem isn't really an API one, it's a great interface. It's that it's implementation is far too clever and would benefit quite a bit from narrowing supported driver styles and removal of some old APIs that really hurt more than help. There could also be better control over opt-in on some behaviors that start adding up in overhead (i.e. change tracking keeps a lot of garbage around).

Re: Rails 3 Performance - Not Good Enough

#113

Post by joevandyk »

stiff wrote: post #111
aneth wrote: post #53

I've been a Rails dev for 5 years. I'm frequently considering leaving for another framework because of one thing: bootstrap time. Starting a test or server on my dev machine takes 20-30 seconds. Particularly with tests, this is a huge problem for doing proper TDD, particularly when you are trying to use tests to track down a bug. In that 20-30 seconds, I often console myself that we no longer need to print punch card…

I have investigated this in detail and the reason for this is really ridiciulous - 80% of this time is spend on executing "require" statements and this is due to really bad design of RubyGems and/or Bundler. They both do their job by augmenting $LOAD_PATH to include all directories containing gem contents. If you then look into how Ruby deals with the $LOAD_PATH, it turns out each time you do a "require" it will go t…

Are you sure that the open calls is the reason why 'require' is so slow?

Re: Rails 3 Performance - Not Good Enough

#114

Post by nirvdrum »

bphogan wrote: post #108
moe wrote: post #49

Earlier quoted context omitted.

culture of not giving a shit about performance Yes. On top of that the ruby ecosystem also seems to attract a certain class of programmers who simply don't know how their magic ruby code translates to CPU, I/O and memory operations. My pet example is a certain popular rails auth framework that hits the database for every single request, to look up tokens that could simply be baked into the cookie. But there are plent…

I've never ever gotten why people get so mad that the database gets hit on every request. It's going to get hit anyway on every request for something else. I've seen how easy it is to break into systems that don't re-auth the user on each request. I guess I just feel like people spend an inordinate amount of time worrying about database performance. A single select from a single table shouldn't be a performance bottl…

Query execution time is only part of the equation. But even that gets ugly fast when you have locking going on. E.g., try adding an index to a non-trivially sized table in MySQL and see how fast that (unnecessary) SELECT performs.

We're a postgres shop and the problem we've run into lately is just the sheer number of open connections to the DB. Using something like PgBouncer helps tremendously, but it still is contingent on there being idle connections. Certainly read slaves could be thrown at the problem, too. But it's mounting complexity for something that isn't all that necessary shrug

Re: Rails 3 Performance - Not Good Enough

#115

Post by joevandyk »

aneth wrote: post #53

I've been a Rails dev for 5 years. I'm frequently considering leaving for another framework because of one thing: bootstrap time. Starting a test or server on my dev machine takes 20-30 seconds. Particularly with tests, this is a huge problem for doing proper TDD, particularly when you are trying to use tests to track down a bug. In that 20-30 seconds, I often console myself that we no longer need to print punch card…

Another person with a similar problem: http://www.ruby-forum.com/topic/393617

Re: Rails 3 Performance - Not Good Enough

#116

Post by stiff »

joevandyk wrote: post #113
stiff wrote: post #111

Earlier quoted context omitted.

I have investigated this in detail and the reason for this is really ridiciulous - 80% of this time is spend on executing "require" statements and this is due to really bad design of RubyGems and/or Bundler. They both do their job by augmenting $LOAD_PATH to include all directories containing gem contents. If you then look into how Ruby deals with the $LOAD_PATH, it turns out each time you do a "require" it will go t…

Are you sure that the open calls is the reason why 'require' is so slow?

I have just written a short blog post about this:

http://www.stifflog.com/2011/05/15/why-is-rails-startup-so-s...

I'm not that deeply in Ruby/RubyGems/Bundler internals to be 100% sure, but on the other hand I really can't see how it could work reasonably fast while using the strategy for looking up libraries it uses.

Re: Rails 3 Performance - Not Good Enough

#117

Post by joevandyk »

aneth wrote: post #53

I've been a Rails dev for 5 years. I'm frequently considering leaving for another framework because of one thing: bootstrap time. Starting a test or server on my dev machine takes 20-30 seconds. Particularly with tests, this is a huge problem for doing proper TDD, particularly when you are trying to use tests to track down a bug. In that 20-30 seconds, I often console myself that we no longer need to print punch card…

I've started a sample Rails 3 application here: https://github.com/joevandyk/slow-rails

Other than a fairly standard list of gems in the Gemfile and one Model with two columns, the application is empty.

On my machine with Ruby 1.9.2-p180, it takes 20 seconds to start. And the application is empty!

Re: Rails 3 Performance - Not Good Enough

#118

Post by jamie_ca »

aneth wrote: post #87
carbon8 wrote: post #82

Earlier quoted context omitted.

"Starting a test or server on my dev machine takes 20-30 seconds. Particularly with tests, this is a huge problem for doing proper TDD" Long startup times or not, sounds like you should probably be using autotest or watchr. 20-30 seconds is not normal. Unless the app has a very large amount of code and dependencies, that sounds like an environment issue. "while most "Rubyists" would rather pollute the namespace" That…

"Long startup times or not, sounds like you should probably be using autotest or watchr. 20-30 seconds is not normal. Unless the app has a massive amount of code and dependencies, that sounds like an environment issue." Autotest helps some, but 20-30 seconds is pretty standard fare. Maybe I use too many gems, but I don't think I should forgo reusing community code to reduce start times. Maybe it's unfair to make a ge…

Polluting the namespace like metawhere does is a significant problem.

We use datamapper on our app, and were looking at doing some of our analytics on mongo at one point, but since both mongomapper and datamapper both define 'helper' methods on Symbols, they can't both be in use at the same time.

If we _really_ wanted to use mongo it would need to be from a completely separate codebase either talking to the db via Sequel or something, or talk to our main app via REST calls - neither being a particularly pleasant solution.

Re: Rails 3 Performance - Not Good Enough

#119

Post by aneth »

stiff wrote: post #111
aneth wrote: post #53

I've been a Rails dev for 5 years. I'm frequently considering leaving for another framework because of one thing: bootstrap time. Starting a test or server on my dev machine takes 20-30 seconds. Particularly with tests, this is a huge problem for doing proper TDD, particularly when you are trying to use tests to track down a bug. In that 20-30 seconds, I often console myself that we no longer need to print punch card…

I have investigated this in detail and the reason for this is really ridiciulous - 80% of this time is spend on executing "require" statements and this is due to really bad design of RubyGems and/or Bundler. They both do their job by augmenting $LOAD_PATH to include all directories containing gem contents. If you then look into how Ruby deals with the $LOAD_PATH, it turns out each time you do a "require" it will go t…

Great work! Yehuda is brilliant, although I'm sure he's busy now and I don't know how much bandwidth he would have to address that. I think he's full bore on Sproutcore.

If I calculate (20 seconds) x (number of times I bootstrap rails), it might even be worth my time to have a go at it.

There is a bug supposed to be fixed in ruby 1.9.3 that addresses something to do with the load path and bootstrap performance, but no idea when that's due to land. If it's truly the cause, it seems like this should be a patch to 1.9.2 instead.

You might also want to see if this is related:

https://github.com/rdp/faster_require/blob/master/lib/faster...

Re: Rails 3 Performance - Not Good Enough

#120

Post by uriel »

aneth wrote: post #72
theli0nheart wrote: post #65

Earlier quoted context omitted.

I haven't been using Rails for nearly as long as you have. As a result, I thought the bootstrap time was something Rails developers just "put up with". I would have thought that someone else would have noticed this problem and done something about it...surprisingly I don't think many have. Django, by comparison, bootstraps the environment almost instantly. I have also been looking for an answer for how to deal with t…

Yeah, I tried Django 2 years ago and sort of wish I had stuck with it primarily for this reason. (Additionally I much prefer the Python "no magic" philosophy, although I've gotten used to tracing through Ruby craziness.) The Rails community (at least at the time) was so much larger and more open that it seemed like a better bet. Rails 3 was coming up and I knew Yehuda was doing brilliant things with the Rails 3.0 arc…

Sadly this days most Python frameworks (including django) are quite full of all kinds of dark magic.

If you want a blissfully magic-free language try Go ( http://golang.org ).

Post reply on HN