Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

axonflux.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#31

Post by charlesju »

I hate when people tell me that Rails doesn't scale because I have scaled Rails and it has nothing to do with the framework.

The reason why websites can't scale is ALWAYS the DB. To fix the DB it is always master/slave, memcached, then sharding. For EVERY language.

Good article, sums up a lot of the tools I used to scale my stuff.

I'd like to add a little more.

1. Turn on slow-query logging in your DB and tail -f the slow query log. Find slow queries and kill them in your code through indices or using several fast queries to make up for 1 long-one.

2. Cache most reads. Both from application caching and memcached.

3. Turn off associations, they don't play well with caching.

4. If you're using memcached, don't use the plugins.

5. If you're a serious startup, use Engine Yard, they're life savers.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#32

Post by davidw »

cperciva wrote: post #17
tptacek wrote: post #16

Earlier quoted context omitted.

Uh, this is how Rails setups work too. You aren't talking directly to Mongrel.

Maybe I misunderstood the article -- it sounded to me like requests were being distributed between Mongrel processes and queued on the individual processes rather than being queued centrally and only allocated to individual processes when a process is free (like Apache does).

The problem with Mongrel is that you allocate N mongrel instances at setup time. Apache, on the other hand, can dynamically allocate new processes (up to a limit) in order to meet increased demand. This is especially important for people like me, who host more than one site on a machine, and want to be able to handle load up to a certain point without fiddling with config files every time there is a spike in traffic.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#33

Post by pcowans »

tptacek wrote: post #4

Great article. Background/deferred job processing has been immensely painful for us, and I have no idea what the accepted best job queue is. We're still stuck on backgroundrb, which is a nightmare. Nanite looks like overkill, and has too many deps.

We're currently using ActiveMQ with ActiveMessaging as the Rails bindings. One reason for this was that we wanted something which would be fairly language/platform neutral if in the future we decided that Ruby/Rails wasn't the right choice for certain bits of the app.

RabbitMQ is another option to consider in this area.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#34

Post by pcowans »

charlesju wrote: post #31

I hate when people tell me that Rails doesn't scale because I have scaled Rails and it has nothing to do with the framework. The reason why websites can't scale is ALWAYS the DB. To fix the DB it is always master/slave, memcached, then sharding. For EVERY language. Good article, sums up a lot of the tools I used to scale my stuff. I'd like to add a little more. 1. Turn on slow-query logging in your DB and tail -f the…

Rails running on vanilla Ruby does have particular scaling issues due to the limitations of the Ruby garbage collector. You hit a GC cycle after every 8MB of allocation, which typically takes around 150ms to run, so can dominate the runtime of your requests. We've had 80% of the runtime in GC, even when you include database time.

This isn't necessarily a killer, but it means there's sometimes more work than you might like in scaling. There are patches to tune the garbage collector, and JRuby etc. may help, but ultimately you need to be much more aware of memory allocation than you might think.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#35

Post by danielrhodes »

charlesju wrote: post #31

I hate when people tell me that Rails doesn't scale because I have scaled Rails and it has nothing to do with the framework. The reason why websites can't scale is ALWAYS the DB. To fix the DB it is always master/slave, memcached, then sharding. For EVERY language. Good article, sums up a lot of the tools I used to scale my stuff. I'd like to add a little more. 1. Turn on slow-query logging in your DB and tail -f the…

Actually, this is generally good advice regardless of what language you're on. The database is almost always at fault. My general rule is that a query should generally take less than 0.001 seconds to execute. If you can get it down to that through proper indexing and whatnot, MySQL will execute the queries fast enough so that they don't get backed up and eventually kill the server. Don't start caching until you can write a proper query. Your site probably doesn't need it if you are writing your queries correctly.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#36

Post by danielrhodes »

100k wrote: post #9

Great post with obvious real world experience behind it. I think it's a fantastic point that you should focus on optimizing your database before you start adding caching. If you can tune your DB with the right indexes and give it enough RAM to fit the whole dataset, you've got a great cache right there! (BTW, I have been using PostgreSQL on my latest project. I'm impressed so far. It has a much query optimizer and be…

Sphinx isn't hard to set up, but (from what I've heard) Solr is a much better choice if you want very fine tuning. Sphinx is a great out of the box solution and will fit most people's needs quite well.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#37

Post by ehedberg »

tptacek wrote: post #4

Great article. Background/deferred job processing has been immensely painful for us, and I have no idea what the accepted best job queue is. We're still stuck on backgroundrb, which is a nightmare. Nanite looks like overkill, and has too many deps.

Huh, I've been quite happy with backgroundrb lately, both for repeating (cron-style) background work and for deferred processing. We've had little trouble tracking worker status once we switched to the memcached-based result cache, which allows the app to interrogate workers however it sees fit...

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#38

Post by wmoxam »

smokinn wrote: post #27
tptacek wrote: post #4

Great article. Background/deferred job processing has been immensely painful for us, and I have no idea what the accepted best job queue is. We're still stuck on backgroundrb, which is a nightmare. Nanite looks like overkill, and has too many deps.

We simply use an in-memory queue. Beanstalkd has worked quite well for us. (Not a rails app though)

Beanstalkd + async_observer (Rails plugin) have worked well for me. Add a Munin script to monitor the queue size (http://gist.github.com/36116) and it's a pretty effective and manageable system.

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#39

Post by reconbot »

tdavis wrote: post #5

I don't know jack about Rails, but there is some good general advise here too. I would have liked to see some DB commentary that didn't choose MySQL as a foregone conclusion. I can't think of many instances where I would recommend it in general.

May I ask what you do use exactly? MySQL has always been my goto for simple db needs. What do you normally use and what's your "general" cases where you wouldn't use it?

Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)

#40

Post by wmoxam »

100k wrote: post #9

Great post with obvious real world experience behind it. I think it's a fantastic point that you should focus on optimizing your database before you start adding caching. If you can tune your DB with the right indexes and give it enough RAM to fit the whole dataset, you've got a great cache right there! (BTW, I have been using PostgreSQL on my latest project. I'm impressed so far. It has a much query optimizer and be…

Have recently switched from using Solr+acts_as_solr to Sphinx+Thinkinh Sphinx I have found quite the opposite. Thinking Sphinx is simpler than Solr and required no 'fiddling'.

I had to work around some problems in acts_as_solr, such as it's tendency to automatically update solr as soon as a DB record changed (regardless if no fields that solr is indexing have changed). Thinking Sphinx simply updates it's index in a batch process that is called by cron (and is super fast!). I highly recommend it.

Post reply on HN