Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's not Ruby that's slow, it's your database

berk.es

Post reply on HN
Print view View original

81–90 of 203 posts

Re: It's not Ruby that's slow, it's your database

#81

Post by JohnBooty »

more_corn wrote: post #19

The default ORM in Ruby… not great at scale. I had a large client with moderate traffic, they were slow because they were absolutely crushing their DB. Now granted it was not JUST ruby, but ruby was a big part of it. We profiled their slow queries and found stuff no human would think is sane. Active record is great for the developer and terrible for database optimization. Which is fine until you try to scale. Then yo…

The question is: is ActiveRecord worse at query generation than other ORMs?

My answer is no, it is not.

In my experience ORMs are all pretty much equal at this. (However, I would love to be wrong!)

I don't know of any ORM that intelligently analyzes your database and comes up with highly optimized queries. I'm not sure such a thing is possible. The database's own query analyzer can optimize query execution because it has intimate knowledge of the database's own data structures, indexes, data cardinality, etc.

I accept ORMs as a necessary evil for most work. However, I enjoy ActiveRecord because unlike some ORMs it makes it fairly easy to simply use raw handwritten SQL when needed.

Re: It's not Ruby that's slow, it's your database

#82

Post by seanhunter »

The biggest database performance problem that I have seen time and time again over the 20+ years I've been using sql databases now is caused by people not understanding that databases are relational (ie designed to perform efficiently over tables, not individual rows) and so running code that essentially causes hundreds/thousands of individual queries where they could just do one big query. Any time you are doing a select query in a loop you should sit down and take a long hard look at your code.

Almost always

  Do big expensive slow query to get everything I need in one hit
  for each row of result
     do something with it
is going to be much faster than

   for each thing in some collection in my program
      get the database row that corresponds to just that one thing
      do something with it
Round trip io, query parse time, cache effects etc are just going to totally destroy the performance of the second one, and optimising the db or programming language perf is going to do almost nothing to improve things if you're doing it this way. Lots of ORM type abstraction layers make it hard sometimes to see that's what you're doing, so people blame db performance but they're actually doing an insanely inefficient algorithm.

My biggest win ever from this was taking an 8 hour operation down to 15s just by moving some sql out of a loop as per the above.

Re: It's not Ruby that's slow, it's your database

#83

Post by byroot »

favorited wrote: post #21
joshmn wrote: post #5

Earlier quoted context omitted.

How many developers will really need it though? It's fast enough for Stripe's API, it's fast enough for Shopify; hardware in exchange for productivity is a pretty fair trade.

Except most companies won't attract the same top-tier Ruby/Rails talent. For example, it looks like 1/3 of the current Rails Core Team work at Shopify, including the person who is the #1 all-time Rails contributor.

Rails Core team member working at Shopify here.

You seem to be misinterpreting what we’re doing at Shopify. We pretty much never work on the product or anything like that, and we don’t go around the app chasing perf issues either.

At best we work on the infra or dev tools to encourage or enforce better patterns, or sometimes in some kind of support capacity for teams having Ruby or Rails specific issues, but Shopify would do fine without us.

We’re mostly here to ensure Rails and the Ruby ecosystem Shopify heavily depends on is maintained and healthy.

Now maybe you need to tier talent to scale a Rails infra (I don’t think so), but Shopify’s presence in the Rails or Ruby core teams is no indication of that.

Re: It's not Ruby that's slow, it's your database

#84

Post by devoutsalsa »

kstrauser wrote: post #75
twotwotwo wrote: post #45

This is kind of relevant now : Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge for server operators used to the levels of traffic from two months ago. Twitter did rewrite their frontend to JVM languages because it was enough of a bottleneck to be worth it. If you can get Mastodon substantially further by serving some views using something…

Ugh, tell me about it. I had an interesting weekend dealing with traffic loads that grew by about 7x in a few days. I wrote about my misadventures at https://blog.freeradical.zone/post/surviving-thriving-throug... . The gist of it is that a RoR “Sidekiq” task queue gets CPU bound after about 25-30 worker threads doing things like making REST API calls to remote servers, querying a database, insert status updates, etc…

Sounds like a job for MUMPS.

Re: It's not Ruby that's slow, it's your database

#88

Post by derefr »

seanhunter wrote: post #82

The biggest database performance problem that I have seen time and time again over the 20+ years I've been using sql databases now is caused by people not understanding that databases are relational (ie designed to perform efficiently over tables, not individual rows) and so running code that essentially causes hundreds/thousands of individual queries where they could just do one big query. Any time you are doing a s…

On the other hand, weird things can happen at scale inside an RDBMS.

For example: window queries, where you're trying to extract `row_number() = 1` of each window-partition. (Or "DISTINCT ON" queries, which are basically syntax sugar for this.) These can be extremely slow, because in many RDBMSes† they generate a query plan that reads+materializes all the rows of each window-partition, in order to then just discard all-but-the-first row-tuple of each window-partition. (In other words, the filtering-out by rank doesn't get pushed back through the query plan to turn the whole-index scan into an index-prefix scan.)

The (surprising) solution to this over-materialization, is to use a recursive CTE, using the previous query's matched key as the inductive case's filter condition, to cause index-prefix matching to happen "one query step at a time." Which, if you think about it, is just "causing hundreds/thousands of individual queries"... just with all of those queries happening within a single tx, statement, query plan, table lock, and (hopefully!) page cursor.

† Some RDBMSes do know how to push this window-node attribute back into the fetch node; the feature is called "index skip scans" or "loose index scans." (Before anyone asks: Postgres isn't one of them... yet. https://wiki.postgresql.org/wiki/Loose_indexscan)

Re: It's not Ruby that's slow, it's your database

#89

Post by kstrauser »

devoutsalsa wrote: post #84
kstrauser wrote: post #75

Earlier quoted context omitted.

Ugh, tell me about it. I had an interesting weekend dealing with traffic loads that grew by about 7x in a few days. I wrote about my misadventures at https://blog.freeradical.zone/post/surviving-thriving-throug... . The gist of it is that a RoR “Sidekiq” task queue gets CPU bound after about 25-30 worker threads doing things like making REST API calls to remote servers, querying a database, insert status updates, etc…

Sounds like a job for MUMPS.

Yeah, I mentioned Elixir in the post as an example of a better alternative.

Re: It's not Ruby that's slow, it's your database

#90

Post by BeefWellington »

Mikeb85 wrote: post #57
likeabbas wrote: post #2

No, Ruby is slow. Its GC is ungodly slow. And you can’t truly multithread, thus you can’t properly parallelize or maximize concurrency.

> No, Ruby is slow. Its GC is ungodly slow. Ruby is quicker than Python and that's the most popular programming language there is at the moment. > And you can’t truly multithread, thus you can’t properly parallelize or maximize concurrency You definitely can. You've been able to fork processes for longer than I can remember. Ractors just became a thing, making multithreading cheaper and easier. Ruby's also had multit…

    Ruby is quicker than Python and that's the most popular programming language there is at the moment.
Do you have any reputable sources for this? All I can find are benchmarks run by companies with clear vested interests in showing one language is faster than the other.

Also, this isn't quite the strongest argument for its performance; popularity hasn't been tied to performance for a long time (see: all those years NodeJS was leading language popularity lists, or Java).

Post reply on HN