Live data from Hacker News

Why your query language should be explicit

blog.hiphipjorge.com

1–10 of 47 posts

Re: Why your query language should be explicit

#2
The biggest problem with a procedural/explicit query is a dynamic system. Without a query planner, you don't have the luxury of a system rewriting your queries. When Table A ~ Table B, but then Table B >> Table A, your queries are going to be radically suboptimal.

Of course, if you're never joining, maybe that's not such a big problem, but you'd have the same issue of a specific range in your table grows disproportionally to another range in the same table and the index you are using is incorrect.

With SQL you can often be very explicit using Common Table Expressions as well in DBs that support it. Otherwise, using subqueries, GROUP BY with HAVING, and several other features often, but not always, prevent radically rewritten queries.

Finally, with SQL on many DBMSs you can still get the explicit wiggle room you need using optimizer hints. No, it's not very portable, but neither is ReQL.

Edit: A few examples of being explicit in SQL (a lá Oracle):

    SELECT /*+ INDEX(name) */ *
      FROM users
      WHERE users.name = 'jorge';


    WITH users_by_name AS (
      SELECT /*+ INDEX(name) */ *
        FROM users
        WHERE users.name = 'jorge'
    )
    SELECT * 
      FROM users_by_name
      JOIN profile using (user_id)
      ORDER BY age;

Re: Why your query language should be explicit

#3
SQL is declarative: you tell the RDBMS what you want and its job is to do it. That's how SQL is and how it's always been. A better RDBMS will optimise the execution plan for you. The point about indexing is somewhat valid, but that's an integral part of schema design and something one should define from the outset based on the data's intended use.

By 'explicit', what I presume the author to mean is 'transparent'. I agree that development processes should be transparent, but I don't necessarily agree that imperative is better than declarative. Indeed, the use case for SQL is data manipulation and analysis; arguably that doesn't come under the remit of 'development', even though programming is involved. Hence the prerequisite of a properly setup schema by someone who knows what they're doing!

Declarative languages definitely have their place.

Re: Why your query language should be explicit

#4
I don't agree with the essay's premise.

>Having your query language be explicit means that you've hit at exactly the right level of abstraction: not too much, but not too little.

I don't like the label of "explicit" as if it's an objective indicator on the continuum between low and high abstraction. It comes across as a value judgement that we'd all agree on and I don't think there's obvious consensus on architecting a data access language.

>If we wanted to dig deeper into this query, we might want to know if the "WHERE" is getting executed before the "ORDER BY". Can we tell from the query if this is the case? No, we can't. You'd have to look it up.

Not knowing the internals of execution is actually a deliberate design feature of SQL. The SQL is meant to be a declarative statement that expresses an algebraic set of rows. (But sometimes, the mathematical purity of this abstraction "leaks" and DBAs/Devs have to add HINTS or do SQL EXPLAIN PLAN to dig into what's happening under the hood -- but that's a separate issue.)

I suppose if one really wanted to affect the order of operations at the SQL syntax level, one could write a VIEW or a subquery with the ORDER BY and then write the outer query with the WHERE clause. I haven't tested this to see if any of the major SQL engines would rewrite this type of convoluted SQL of ORDER BY -then- WHERE clause.

Yes, with UNIX command line, you have different execution characteristics of "ls | grep | sort" vs "ls | sort | grep" but one can't translate that explicit-sequence-of-execution mental model to SQL.

* Does this increase cognitive load? Yes, it does. But this is outweighed by the ability to understand how your query is being executed.

I'm not convinced of this conclusion.

Also, I'm not sure RethinkDB works like this as a deliberately engineered advantage. The RethinkDB devs can clarify but it's possible for their engine to work like this because it's more straightforward to implement the parser and not because there is overwhelming inherent superiority to this approach. With traditional SQL (e.g. Oracle, MSSQL, etc), the query rewriting engines are very mature and can be more aggressive and thereby fulfilling the goal of a declarative mathematical purity. However, it takes lots of programmer man-hours to translate declarative SQL into optimal execution plans.

Re: Why your query language should be explicit

#5
I don't agree with this conclusion. I've worked with Postgres, MySQL and Oracle, and found that it is important to have a good understanding of the possible execution plans for a query. I will sometimes construct a query very carefully to achieve a particular plan. And when things go wrong, I run EXPLAIN PLAN, examine statistics, etc., and tweak my query to do what I want. You really have to do that to obtain good performance in some cases. It does undercut the claim that SQL has to being non-procedural, but that's life.

I do NOT want to have to construct the query execution plan manually for every single query. Usually, the optimizer will do a fine job, assuming the database designer has made good choices regarding indexes and other physical design issues. But there are always a few complex, tricky queries where you do have to understand internals. The "cognitive load" is the same, and having a high-level query language means that when you do make your subtle and deft change to rescue performance, you often just tweak a query, instead of rewriting a detailed query plan.

(Extending this position a little: I have come around 180 degrees and consider ORMs to be a bad idea overall. When you do need to be very careful about writing a query, the ORM adds a layer of complexity -- not only do you need to control the SQL, but you now need to ensure that your ORM can produce that SQL.)

Re: Why your query language should be explicit

#6
post #2

The biggest problem with a procedural/explicit query is a dynamic system. Without a query planner, you don't have the luxury of a system rewriting your queries. When Table A ~ Table B, but then Table B >> Table A, your queries are going to be radically suboptimal. Of course, if you're never joining, maybe that's not such a big problem, but you'd have the same issue of a specific range in your table grows disproportio…

Even if you're not doing any joins, the selectivity of your data matters. It's more efficient to perform a table scan than using an index if the selectivity of your filter is low. I wouldn't want to have to manually rewrite a procedural query when my data changes.

I feel like this is one of those "X is categorically better than Y" or "X is better than Y because Z" articles. The problem with these types of articles is that they tend to take an extreme position when the reality is much more nuanced. For the subset of relatively simple problems the author has encountered, perhaps the procedural query is easier to reason about, but there are vastly complex SQL queries that wouldn't benefit from an explicit query like this.

Re: Why your query language should be explicit

#7
post #5

I don't agree with this conclusion. I've worked with Postgres, MySQL and Oracle, and found that it is important to have a good understanding of the possible execution plans for a query. I will sometimes construct a query very carefully to achieve a particular plan. And when things go wrong, I run EXPLAIN PLAN, examine statistics, etc., and tweak my query to do what I want. You really have to do that to obtain good pe…

In theory, if you had an explicit query language, you could write the query planner in that language. You could then rely on the default query plan by calling functions invoking it, or make part or all of that plan explicit by invoking the lower-level functions directly.

Re: Why your query language should be explicit

#9
post #6
post #2

The biggest problem with a procedural/explicit query is a dynamic system. Without a query planner, you don't have the luxury of a system rewriting your queries. When Table A ~ Table B, but then Table B >> Table A, your queries are going to be radically suboptimal. Of course, if you're never joining, maybe that's not such a big problem, but you'd have the same issue of a specific range in your table grows disproportio…

Even if you're not doing any joins, the selectivity of your data matters. It's more efficient to perform a table scan than using an index if the selectivity of your filter is low. I wouldn't want to have to manually rewrite a procedural query when my data changes. I feel like this is one of those "X is categorically better than Y" or "X is better than Y because Z" articles. The problem with these types of articles is…

Yeah that's definitely true, I tried to allude to that with the ranges example, "Elvis" would be good for an index, whereas "Chris" or "Matt" you might as well do a full table scan.
Post reply on HN