Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
1–10 of 56 posts
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#2Pretty good run down. Actually, a very good run down.
Sphinx is cool, but there's no reason to be afraid of solr/lucene. It takes _very_ little java knowledge to get it up and running and it's very very, really, crazy fast. Like, hundreds of thousands of searches a day on millions of documents and it's totally stable. knocks on wood
Also, Passenger is indeed better than all this mongrel + god + tweaks they're talking about it. AboutUs.org (my employeer) is the largest site on Passenger (that we can find) and we've had 2 actual crashes in the last 6 months, and they were fixed with rebooting Apache.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#3Pretty good run down. Actually, a very good run down. Sphinx is cool, but there's no reason to be afraid of solr/lucene. It takes _very_ little java knowledge to get it up and running and it's very very, really, crazy fast. Like, hundreds of thousands of searches a day on millions of documents and it's totally stable. knocks on wood Also, Passenger is indeed better than all this mongrel + god + tweaks they're talking…
Thanks for the suggestions! Will probably explore Solr down the road. I've written search on top of Lucene before, just didn't have time to work too hard on getting search going in between all the other features we're trying to push out.
Passenger I have heard great things -- been meaning to try since it does fix the mongrel queue problem.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#4Great article.
Background/deferred job processing has been immensely painful for us, and I have no idea what the accepted best job queue is. We're still stuck on backgroundrb, which is a nightmare. Nanite looks like overkill, and has too many deps.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#5I don't know jack about Rails, but there is some good general advise here too. I would have liked to see some DB commentary that didn't choose MySQL as a foregone conclusion. I can't think of many instances where I would recommend it in general.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#6Great article. Background/deferred job processing has been immensely painful for us, and I have no idea what the accepted best job queue is. We're still stuck on backgroundrb, which is a nightmare. Nanite looks like overkill, and has too many deps.
I'm 'stuck' on backgroundrb too. Until now I thought it was the default and best way to run processes in the b/g for a Rails site.
I'm eager to see if anyone else can recommend alternatives that they've had good experiences with.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#7Great article. Background/deferred job processing has been immensely painful for us, and I have no idea what the accepted best job queue is. We're still stuck on backgroundrb, which is a nightmare. Nanite looks like overkill, and has too many deps.
Workling has worked well for us -- Rany Keddo is a phenomenal open source guy.
Tobi's deferred jobs also looked totally solid to me. That one's proven because it drives Shopify. We use his liquid plugin, which is also stellar.
That underlines a big issue with Rails dev even today -- its really hard to know what's good / what works, and what is just some weekend project for someone.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#8Pretty good run down. Actually, a very good run down. Sphinx is cool, but there's no reason to be afraid of solr/lucene. It takes _very_ little java knowledge to get it up and running and it's very very, really, crazy fast. Like, hundreds of thousands of searches a day on millions of documents and it's totally stable. knocks on wood Also, Passenger is indeed better than all this mongrel + god + tweaks they're talking…
From my experience (and I have used Sphinx, Lucene, Solr and Xapian in production) is that Lucene/Solr have a pretty bad perfomance compared to Sphinx or Xapian.
My Lucene setup began to throw deadlocks and memory exceptions pretty early on. Searching on "deadlock lucene" on Google yields 25000 results. I have later rewritten the system to Xapian where it has run without any problems.
For live updates I would recommend using Xapian. For fairly static indexes I would recommend Sphinx (as it's _really_ fast for both indexing and searching, but it does not support live index updates yet).
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#9Great post with obvious real world experience behind it.
I think it's a fantastic point that you should focus on optimizing your database before you start adding caching. If you can tune your DB with the right indexes and give it enough RAM to fit the whole dataset, you've got a great cache right there!
(BTW, I have been using PostgreSQL on my latest project. I'm impressed so far. It has a much query optimizer and better indexes than MySQL.)
I also like using Solr/acts_as_solr. I haven't used Sphinx but from what I've read about setting it up it sounds incredibly fiddly. Solr, by contrast, is quite simple.
Re: Building and Scaling a Startup on Rails: Things We Learned the Hard Way (by Posterous S08)
#10Pretty good run down. Actually, a very good run down. Sphinx is cool, but there's no reason to be afraid of solr/lucene. It takes _very_ little java knowledge to get it up and running and it's very very, really, crazy fast. Like, hundreds of thousands of searches a day on millions of documents and it's totally stable. knocks on wood Also, Passenger is indeed better than all this mongrel + god + tweaks they're talking…
From my experience (and I have used Sphinx, Lucene, Solr and Xapian in production) is that Lucene/Solr have a pretty bad perfomance compared to Sphinx or Xapian. My Lucene setup began to throw deadlocks and memory exceptions pretty early on. Searching on "deadlock lucene" on Google yields 25000 results. I have later rewritten the system to Xapian where it has run without any problems. For live updates I would recomme…
Really? What size of index/documents where you doing searches on?
We're doing live updating, probably close to 1 update per second.