Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's not Ruby that's slow, it's your database

berk.es

Post reply on HN
Print view View original

71–80 of 203 posts

Re: It's not Ruby that's slow, it's your database

#71

Post by JohnBooty »

WkndTriathlete wrote: post #23
lmm wrote: post #10

Earlier quoted context omitted.

There is always an alternative to writing SQL queries by hand, and it's usually a better one IME. Any ORM worth its salt will let you do the query in your blog post via the ORM, as a single query.

> There is always an alternative to writing SQL queries by hand, and it's usually a better one IME. I spent years writing code using Spring/Hibernate, and I can state with certainty that both of those statements are demonstrably false. Every application starts with good intentions, a simple CRUD webapp, and an ORM, then at some point the business requirements yield an N+1 problem in ORMs or several non-trivial left j…

I agree with you, however, some ORMs handle this fairly elegantly.

    At that point it's far easier to write the query in 
    straight SQL and produce a straightforward mapping 
    into the record structure, which doesn't play well 
    with the ORM because that bypasses its entity cache, 
    which causes another huge set of problems on its own
Rails' ActiveRecord ORM offers at least two ways to handle this.

1. ActiveRecord plays really nicely with views (including materialized views) in my experience. It treats them just like tables, basically, except you can't write to them. (note: there may actually be some cases where you can write to them; not sure)

2. You can supply your own handrolled SQL to ActiveRecord, e.g. `User.find_by_sql("select a,b,c from blahblahblah")`

YMMV obviously but I've been working with Rails since 2014 but this has covered all of my performance needs.

Plain old ActiveRecord default query generation is fine 99% of the time, and it's rather elegant/easy to sidestep it when I wish.

Re: It's not Ruby that's slow, it's your database

#72

Post by JohnBooty »

rahimnathwani wrote: post #43
int_19h wrote: post #39

Earlier quoted context omitted.

A pure query builder is just writing SQL on syntax tree level, more or less. This makes sense for the same reason why you want your macros to operate on ASTs and not raw text. But I would argue that it's still much closer to plain text SQL in the code than to any ORM.

Right, but it lives with your application code and has the same syntax as the application code. That's probably preferable to SQL stored procedures (which often live outside source control).

    That's probably preferable to SQL stored 
    procedures (which often live outside source control).
Stored procs definitely have some big pros and big cons, but I don't think this is one of them -- any ORM with a decent set of tools to manage migrations (ActiveRecord is one) makes this objection a non issue IMO.

Re: It's not Ruby that's slow, it's your database

#74

Post by JohnBooty »

clintonb wrote: post #54
joshmn wrote: post #5

Earlier quoted context omitted.

How many developers will really need it though? It's fast enough for Stripe's API, it's fast enough for Shopify; hardware in exchange for productivity is a pretty fair trade.

> It's fast enough for Stripe's API... Stripe is moving to Java.

If you need maximum requests per second for sufficiently high volumes of traffic - ie, Stripe/Twitter/FB/whatever stuff - then yeah Ruby/Python/etc doesn't cut it.

If your needs get extreme enough (high frequency trading?) then Java doesn't cut it either.

Most (I suspect > 99%) use cases don't fit these criteria. I know this is skewed a perception is skewed a little by the HN crowd, where many folks legitimately are trying to build the next Stripe or Twitter, but I think the world of modern software development is somewhat poisoned by this belief that we all need to follow FAANG-scale practices and architectures.

Re: It's not Ruby that's slow, it's your database

#75

Post by kstrauser »

twotwotwo wrote: post #45

This is kind of relevant now : Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge for server operators used to the levels of traffic from two months ago. Twitter did rewrite their frontend to JVM languages because it was enough of a bottleneck to be worth it. If you can get Mastodon substantially further by serving some views using something…

Ugh, tell me about it. I had an interesting weekend dealing with traffic loads that grew by about 7x in a few days. I wrote about my misadventures at https://blog.freeradical.zone/post/surviving-thriving-throug... .

The gist of it is that a RoR “Sidekiq” task queue gets CPU bound after about 25-30 worker threads doing things like making REST API calls to remote servers, querying a database, insert status updates, etc. I can’t help but to think that the equivalent written in Go, or even Ruby-without-Rails, could handle many times the traffic with fewer CPU and RAM resources.

Re: It's not Ruby that's slow, it's your database

#76

Post by int_19h »

nicoburns wrote: post #56
rahimnathwani wrote: post #43

Earlier quoted context omitted.

Right, but it lives with your application code and has the same syntax as the application code. That's probably preferable to SQL stored procedures (which often live outside source control).

I’ve found that all this does is make the query less readable. SQL is purpose made for writing queries, and avoids unnecessary syntax noise you get when trying to fit the query into a host language based dsl.

That really depends on the language - specifically, on whether it already has constructs that can map nicely (e.g. LINQ in C#), or macros to define them, or syntax that is generally amenable to DSLs even without macros in the picture (e.g. Lisps).

SQL itself is also not a particularly well-designed query language. E.g. the order of the query doesn't reflect the natural data flow (SELECT .. FROM .. is reversed - compare to XQuery's FLWOR, for example), there are warts like WHERE vs HAVING etc. A good DSL can do much better.

Re: It's not Ruby that's slow, it's your database

#77

Post by lostcolony »

ColonelPhantom wrote: post #62
riwsky wrote: post #42

Earlier quoted context omitted.

They’re out there—but I just ran EXPLAIN ANALYZE on my company’s search for them, and the query planner’s not too happy. It has to scan the entire resume heap before doing an on-site join with candidates, all while dealing with resource contention from queries of other employers. I know recruiters and e.g. triplebyte, stackoverflow all offer indexes to speed up this search, but in our case those indexes wouldn’t fit…

Why does the index need to fit in "cash" (cache RAM)? Generally an index on disk, especially SSD, is far faster than traversing the entire dataset, because it allows the query executor to quickly narrow it down. Even if this requires some disk IO, it's a lot faster than doing all the disk IO for the entire dataset.

I'm not sure if you missed the joke (he basically was saying "It's hard to hire people who can do this sort of thing; they're out there but in demand and recruiters are expensive" but using DB terminology) and are thinking he's actually describing a DB operation, or if you're building on it so obliquely -I- am missing the joke you're trying to make.

Re: It's not Ruby that's slow, it's your database

#78

Post by JohnBooty »

jojobas wrote: post #8

From my experience with Python and Ruby, it's not the database itself, it's the database driver that spends quite some time on serialization and deserialization every time you want to send around some data, even more so when you pack some objects to store as semistructured/json/xml. So no, it's not the database, it's your interpreted language.

Before we proceed, are you aware that a lot of popular database drivers for Ruby (and Python? not sure) implement the performance-critical bits in good old natively compiled C?

For example, the Ruby postgres gem: https://github.com/ged/ruby-pg/tree/master/ext

(I wasn't sure until I checked just now, so I'm not questioning your familiarity with the tech. Just not sure if that's commonly known)

    So no, it's not the database, it's your interpreted language.
Or it's your developers.

If you're moving enough data over the wire that application-side serialization/deserialization becomes an issue, often this is because developers are retrieving a whole truckload of records when they really only ought to be returning one.

Even in a fast application language, this is a problem. You're still burning extra CPU cycles, you're still allocating extra RAM, you're moving the same amount of extra data "over the wire", and you're still consuming extra database resources.

Of course, there truly are many use cases where you might want to move a truckload of data between your application and your database, and do some heavy crunching on the application side. I wholeheartedly agree Ruby is not ideal for that.

Re: It's not Ruby that's slow, it's your database

#79

Post by JohnBooty »

rubyist5eva wrote: post #50

I’m a Rails developer by trade, been doing it for over a decade. Bad indexing and lazy N+1 are 90% of the performance problems in a typical rails application. The other 10% is when people do aggregates or joins in application code instead of sql. I absolutely despise ActiveRecord because it makes non-trivial aggregates a pain to write. Sequel is a much better ORM but good luck getting a team on board.

    I absolutely despise ActiveRecord because it makes 
    non-trivial aggregates a pain to write
I'm a SQL guy at heart.

I certainly agree with you... I loathe writing any non-trivial query in ActiveRecord's query builder.

However, I find ActiveRecord does a really good job of getting out of the way when I deem it easier to write some raw SQL.

Generally do one of two things.

If I have a really big nasty query, I'll implement it as a (standard or materialized) database view. I may query it directly or map a model to it.

Or, I'll just use `MyModel.find_by_sql("blahblahblah")` which happily accepts my handwritten SQL.

I don't have a link handy but DHH stated ages ago that the ability to do stuff like that was an explicit goal of ActiveRecord.

Re: It's not Ruby that's slow, it's your database

#80

Post by sa46 »

boloust wrote: post #60

I was left confused after reading this article. 1. It claims Rust is ~10x faster than Ruby, based on a benchmark that reads a 23mb file, and then iterates over the data a single time. In my experience, Rust is between 20-100x faster than Ruby in purely CPU-bound workloads. But the author's main contention is that most work is IO-bound instead of CPU-bound, so probably not a big deal. 2. The author claims "it hardly m…

I'm also confused.

2. Agreed. 150 ms is extraordinarily slow for a point lookup from a single table. A simple lookup should take around 1 ms since it'll be cached in memory.

3. Also agreed. The actual database time looks to be the first flame. Hovering over it shows PG::Connection::exec which accounts for 2.5% of the time.

I was curious about date parsing and dug up the source [1]. Seems like you could gain a ton of speed back by using a postgres specific timestamp parsing routine. In Go, it's 40 lines [2]

[1] https://github.com/ruby/date/blob/d21c69450a57a1931ab6190385...

[2]: https://github.com/jackc/pgx/blob/a968ce3437eefc4168b39bbc4b...

Post reply on HN