Efficient GraphQL Queries in Ruby on Rails and Postgres • Hacker News Board

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#51

Post by jblwps » Thu, Sep 26, 2019, 1:42 AM UTC

uhoh-itsmaciek wrote: post #32

e12e wrote: post #13

Somewhat on the side, but: > By using the includes method we've been able to knock our queries down from six to two: The first to find the events, and the second to find the categories for those events. Am I the only one that thinks rails/activerecord is crazy here for (often) preferring two trips to the db over a join on indexed primary keys? Are there really (sane) use-cases where that is better?

>Am I the only one that thinks rails/activerecord is crazy here for (often) preferring two trips to the db over a join on indexed primary keys? You mean SELECT u.* , c.* FROM users u INNER JOIN comments c ON u.id = c.user_id WHERE u.id = ? If the average user record includes a non-trivial amount of data and there are many comments, you're repeating a lot of information in the query results. disclaimer: I'm currently…

Is the concern over repeated information about [load on the DB], [size of data sent to the client], or something like [processing requirements on the client]? I'm assuming you're talking about [size of data sent to the client], but shouldn't that data be readily-compressible?

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#52

Post by earthboundkid » Thu, Sep 26, 2019, 2:19 AM UTC

m12k wrote: post #35

xyzzy_plugh wrote: post #2

> The following query produces the data that we want... we just need to figure out how to write a batch loader that generates the same result. This is where the abstraction breaks down. Should you care what the query ultimately looks like? For anything beyond a toy, you sure should. But how far should you go to ensure the abstraction produces the implementation you want? Why not just skip to the implementation you wa…

I've seen many mobile developers be frustrated at constantly needing to ask the backend developers to make new rest endpoints for them because the existing ones were not exposing the data in the way they needed it exposed. And similarly I've seen backend devs be frustrated at continually having to make new endpoints for those nagging mobile devs. If you have teams working independently of one another, where the backe…

It seems that the problem that GraphQL solves is Conway’s Law. For an org that can keep the backend and frontend together, it’s not much of an improvement.

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#53

Post by xrd » Thu, Sep 26, 2019, 3:42 AM UTC

earthboundkid wrote: post #52

m12k wrote: post #35

Earlier quoted context omitted.

I've seen many mobile developers be frustrated at constantly needing to ask the backend developers to make new rest endpoints for them because the existing ones were not exposing the data in the way they needed it exposed. And similarly I've seen backend devs be frustrated at continually having to make new endpoints for those nagging mobile devs. If you have teams working independently of one another, where the backe…

It seems that the problem that GraphQL solves is Conway’s Law. For an org that can keep the backend and frontend together, it’s not much of an improvement.

I would argue that I've never seen even small teams keep the backend and frontend together if they are doing anything important. GraphQL is, if nothing else, a nice layer over your data that allows the backend developers to see what the front end developer are doing better than REST. That introspection is really helpful.

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#54

Post by rytill » Thu, Sep 26, 2019, 4:24 AM UTC

therusskiy wrote: post #3

to solve N+1 in Ruby Graphql automatically I wrote a thingy that analyzes what fields were requested in a recursive manner and then uses a preloader at the root level to load everything in advance. Solve 80% of N+1 problems automatically.

Don't know why this isn't builtin to GQL libraries - seems like the whole point of GQL is to do these types of queries and it's horribly inefficient by default.

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#55

Post by uhoh-itsmaciek » Thu, Sep 26, 2019, 4:52 AM UTC

jblwps wrote: post #51

uhoh-itsmaciek wrote: post #32

Earlier quoted context omitted.

>Am I the only one that thinks rails/activerecord is crazy here for (often) preferring two trips to the db over a join on indexed primary keys? You mean SELECT u.* , c.* FROM users u INNER JOIN comments c ON u.id = c.user_id WHERE u.id = ? If the average user record includes a non-trivial amount of data and there are many comments, you're repeating a lot of information in the query results. disclaimer: I'm currently…

Is the concern over repeated information about [load on the DB], [size of data sent to the client], or something like [processing requirements on the client]? I'm assuming you're talking about [size of data sent to the client], but shouldn't that data be readily-compressible?

I mostly meant data size on the wire, yeah, but these are all potential issues. In any case, the performance difference between the two approaches is going to heavily depend on your schema and your data, and I suspect the AR works well enough in practice.

And in this example, the data is definitely compressible, but the Postgres wire protocol does not implement compression (it leaves that to tls right now, though CRIME makes that a problem). Adding native compression to the protocol has been on the TODO list for a while: https://wiki.postgresql.org/wiki/Todo#Wire_Protocol_Changes_...

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#56

Post by ako » Thu, Sep 26, 2019, 5:21 AM UTC

atombender wrote: post #41

xyzzy_plugh wrote: post #2

> The following query produces the data that we want... we just need to figure out how to write a batch loader that generates the same result. This is where the abstraction breaks down. Should you care what the query ultimately looks like? For anything beyond a toy, you sure should. But how far should you go to ensure the abstraction produces the implementation you want? Why not just skip to the implementation you wa…

The problem with "just SQL" is that you arguably want to compose queries. A foundational observation that GraphQL makes is that every query is different depending on the client's needs. You can do "SELECT * FROM users JOIN companies ON companies.id = users.employer_id" and call it a day, but now you're already probably overfetching, and the next evolutions of your codebase ends up accruing extra logic to add or remov…

Or you can use OData, which has the same benefits as graphql.

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#57

Post by [deleted] » Thu, Sep 26, 2019, 5:54 AM UTC

xyzzy_plugh wrote: post #2

> The following query produces the data that we want... we just need to figure out how to write a batch loader that generates the same result. This is where the abstraction breaks down. Should you care what the query ultimately looks like? For anything beyond a toy, you sure should. But how far should you go to ensure the abstraction produces the implementation you want? Why not just skip to the implementation you wa…

[deleted]

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#58

Post by Deimorz » Thu, Sep 26, 2019, 6:07 AM UTC

sandstrom wrote: post #30

We've looked at GraphQL but ended up going with Graphiti instead. Only a few weeks since we implemented, but happy so far. Feels like it's the best of REST and GraphQL. This page from Graphiti[1] explains the thinking behind it. Don't be discouraged by the loud pink colors ;) [1] https://www.graphiti.dev/guides/why

Thanks - I hadn't heard of Graphiti before, but it looks really interesting. I like a lot of the aspects of its design and capabilities, I'll definitely need to read more about it.

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#59

Post by [deleted] » Thu, Sep 26, 2019, 8:34 AM UTC

atombender wrote: post #41

xyzzy_plugh wrote: post #2

> The following query produces the data that we want... we just need to figure out how to write a batch loader that generates the same result. This is where the abstraction breaks down. Should you care what the query ultimately looks like? For anything beyond a toy, you sure should. But how far should you go to ensure the abstraction produces the implementation you want? Why not just skip to the implementation you wa…

The problem with "just SQL" is that you arguably want to compose queries. A foundational observation that GraphQL makes is that every query is different depending on the client's needs. You can do "SELECT * FROM users JOIN companies ON companies.id = users.employer_id" and call it a day, but now you're already probably overfetching, and the next evolutions of your codebase ends up accruing extra logic to add or remov…

[deleted]

Re: Efficient GraphQL Queries in Ruby on Rails and Postgres

#60

Post by nicoburns » Thu, Sep 26, 2019, 10:04 AM UTC

Izkata wrote: post #40

tyre wrote: post #36

Earlier quoted context omitted.

I agree with everything except the five seconds note. Web pages loading in fives seconds is an awful experience. Optimizing expensive queries not only improves the user experience (be it web or api or whatever) but also can open up entire usecases that otherwise wouldn’t be practical. Five second queries are a primate candidate for DOS attacks, whether intentionally malicious or by users spamming refresh, retry, etc.

I think he meant shaving only 5 seconds off of the 200 second query.

Maybe, but shaving almost 5 seconds off a 5 second query can be quite realistic for some pathological ORM queries. I use ORMs for the simple cases, but it's definitely worth cutting them out in the complex ones.