It's not Ruby that's slow, it's your database • Hacker News Board

Re: It's not Ruby that's slow, it's your database

#91

Post by KingOfCoders » Wed, Nov 09, 2022, 6:01 AM UTC

I do think this might have been true in the past, but SSDs have changed the game for databases. Databases became 10x faster over night but this hasn't sunk in to every developer yet.

Also ORMs have created an environment, where bad queries are sent to the database for questionable gains (after saving time writing some initial code). After writing a nice Java ORM in the 90s myself (also wrote my own Ruby ORM framework called Ruby.RO before Rails became a thing), then using Hibernate/JPA extensively for a decade and later ORMs in Scala and TS.

I now prefer plain SQL (with Go). Quite often you can write some CTE that is fast but would have been several calls by an ORM. I also now use CTE to read/update/deleate data in one query (but be careful b/c execution order).

Re: It's not Ruby that's slow, it's your database

#92

Post by Jensson » Wed, Nov 09, 2022, 6:02 AM UTC

adenozine wrote: post #87

anonygler wrote: post #86

It’s definitely ruby that’s slow. Your database may also be slow. I love ruby. But it’s slow. It’s a great choice for a lot of things.

Got some sources? Ruby3 was faster than python at some point, iirc

Python is slow.

Re: It's not Ruby that's slow, it's your database

#93

Post by hamandcheese » Wed, Nov 09, 2022, 6:06 AM UTC

kstrauser wrote: post #75

twotwotwo wrote: post #45

This is kind of relevant now : Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge for server operators used to the levels of traffic from two months ago. Twitter did rewrite their frontend to JVM languages because it was enough of a bottleneck to be worth it. If you can get Mastodon substantially further by serving some views using something…

Ugh, tell me about it. I had an interesting weekend dealing with traffic loads that grew by about 7x in a few days. I wrote about my misadventures at https://blog.freeradical.zone/post/surviving-thriving-throug... . The gist of it is that a RoR “Sidekiq” task queue gets CPU bound after about 25-30 worker threads doing things like making REST API calls to remote servers, querying a database, insert status updates, etc…

Did you actually observe CPU at 100%, or just that your sidekiq workers were falling behind?

Solid chance that you are still IO bound and could push past 25 workers. The main bottleneck is probably RAM.

Re: It's not Ruby that's slow, it's your database

#94

Post by Jensson » Wed, Nov 09, 2022, 6:06 AM UTC

JohnBooty wrote: post #78

jojobas wrote: post #8

From my experience with Python and Ruby, it's not the database itself, it's the database driver that spends quite some time on serialization and deserialization every time you want to send around some data, even more so when you pack some objects to store as semistructured/json/xml. So no, it's not the database, it's your interpreted language.

Before we proceed, are you aware that a lot of popular database drivers for Ruby (and Python? not sure) implement the performance-critical bits in good old natively compiled C? For example, the Ruby postgres gem: https://github.com/ged/ruby-pg/tree/master/ext (I wasn't sure until I checked just now, so I'm not questioning your familiarity with the tech. Just not sure if that's commonly known) So no, it's not the data…

Marshalling the result into python or ruby objects is expensive no matter how you do it, and that cost is due to the programming language you need to marshal to, the logic being implemented in C can't help that.

Re: It's not Ruby that's slow, it's your database

#95

Post by hamandcheese » Wed, Nov 09, 2022, 6:12 AM UTC

JohnBooty wrote: post #72

rahimnathwani wrote: post #43

Earlier quoted context omitted.

Right, but it lives with your application code and has the same syntax as the application code. That's probably preferable to SQL stored procedures (which often live outside source control).

That's probably preferable to SQL stored procedures (which often live outside source control). Stored procs definitely have some big pros and big cons, but I don't think this is one of them -- any ORM with a decent set of tools to manage migrations (ActiveRecord is one) makes this objection a non issue IMO.

I explicitly do not want to manage stored procedures in the same way as typical migrations - if I did, I would wind up with many, many versions of the procedure in my code base as it evolved over time. This would make grepping or locating the latest version pretty annoying.

Flyway (migration tool in Java) has a notion of “repeatable” migrations, though, which would do the trick.

Re: It's not Ruby that's slow, it's your database

#96

Post by wonnage » Wed, Nov 09, 2022, 6:17 AM UTC

boloust wrote: post #60

I was left confused after reading this article. 1. It claims Rust is ~10x faster than Ruby, based on a benchmark that reads a 23mb file, and then iterates over the data a single time. In my experience, Rust is between 20-100x faster than Ruby in purely CPU-bound workloads. But the author's main contention is that most work is IO-bound instead of CPU-bound, so probably not a big deal. 2. The author claims "it hardly m…

That’s covered on literally the line after the chart:

> The parsing (juggling of data) takes the majority of time: DateTime::parse

Re: It's not Ruby that's slow, it's your database

#97

Post by gedy » Wed, Nov 09, 2022, 6:23 AM UTC

knodi123 wrote: post #66

gedy wrote: post #25

Earlier quoted context omitted.

Yeah antidotal, but I was doing some tests of an existing Rails app API compared to Hasura (erlang GraphQL server) talking to same DB, and querying same data was like 100x faster in Hasura..

anecdotal: based on personal experience antidotal: based on curing a deadly poison

Thanks, love what autocorrect comes up with..

Re: It's not Ruby that's slow, it's your database

#98

Post by kstrauser » Wed, Nov 09, 2022, 6:26 AM UTC

hamandcheese wrote: post #93

kstrauser wrote: post #75

Earlier quoted context omitted.

Ugh, tell me about it. I had an interesting weekend dealing with traffic loads that grew by about 7x in a few days. I wrote about my misadventures at https://blog.freeradical.zone/post/surviving-thriving-throug... . The gist of it is that a RoR “Sidekiq” task queue gets CPU bound after about 25-30 worker threads doing things like making REST API calls to remote servers, querying a database, insert status updates, etc…

Did you actually observe CPU at 100%, or just that your sidekiq workers were falling behind? Solid chance that you are still IO bound and could push past 25 workers. The main bottleneck is probably RAM.

I actually saw them at 100% CPU with RAM to spare.

Re: It's not Ruby that's slow, it's your database

#99

Post by JohnBooty » Wed, Nov 09, 2022, 6:35 AM UTC

hamandcheese wrote: post #95

JohnBooty wrote: post #72

Earlier quoted context omitted.

That's probably preferable to SQL stored procedures (which often live outside source control). Stored procs definitely have some big pros and big cons, but I don't think this is one of them -- any ORM with a decent set of tools to manage migrations (ActiveRecord is one) makes this objection a non issue IMO.

I explicitly do not want to manage stored procedures in the same way as typical migrations - if I did, I would wind up with many, many versions of the procedure in my code base as it evolved over time. This would make grepping or locating the latest version pretty annoying. Flyway (migration tool in Java) has a notion of “repeatable” migrations, though, which would do the trick.

    This would make grepping or locating the latest version pretty annoying

Wouldn't this be an issue with any database object managed via migrations? Do any of them make this easy for any database object?

In ActiveRecord, you have your migrations folder(s) and then you have your `structure.sql` (essentially the raw output of mysqldump or pgdump) or the equivalent.

If I need to see the literal database definition of any database object I look it up in there. Not the slickest solution but works well enough - really just a few keystrokes in my editor.

I'd be curious how other migration tools handle (or fail to handle) this.

Re: It's not Ruby that's slow, it's your database

#100

Post by tinus_hn » Wed, Nov 09, 2022, 6:39 AM UTC

twotwotwo wrote: post #45

This is kind of relevant now : Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge for server operators used to the levels of traffic from two months ago. Twitter did rewrite their frontend to JVM languages because it was enough of a bottleneck to be worth it. If you can get Mastodon substantially further by serving some views using something…

Twitter spent a billion a year running their service, surely a lot of that can be shaved off but I think it’s a bit simplistic to think people can just run a comparable service for free.