Live data from Hacker News

SQL Tips and Tricks

github.com

81–90 of 168 posts

Re: SQL Tips and Tricks

#81

I'll add some of mine: Learn your DB server. Check the query plans often. You might get surprised. Tweak and recheck. Usually EXISTS is faster than IN. Beware that NOT EXISTS behaves differently than EXCEPT in regards to NULL values. Instead of joining tables and using distinct or similar to filter rows, consiser using subquery "columns", ie in SELECT list. This can be much faster even if you're pulling 10+ values fr…

Instead of joining tables and using distinct or similar to filter rows, consiser using subquery "columns", ie in SELECT list. What does this mean? Running SELECT column1, ( SELECT column2, column3, ... FROM table_b WHERE table_a.id = table_b.a_id ) FROM table_a Results in "subquery must return only one column" as I expected. You mean returning the multiple columns as a record / composite type? Keep in mind GROUP BY c…

> > Keep in mind GROUP BY clause usually dictates index use.

> The reason for this wasn't immediately apparent to me.

The key thing to remember is that grouping is essentially a sorting operation, and it happens before your other sorts (that last part isn't necessarily as obvious).

Re: SQL Tips and Tricks

#82

Earlier quoted context omitted.

What about `WHERE true`?

I don't understand the point at all. If you need to add some condition later on, why not just add it then? What benefit is there to just marking out the spot where you might add the condition at some point in the future?

I personally don't use it too, but I think it's origins are not just readability, but from developing queries in a REPL like environment.

As you develop and are constantly creating / debugging queries where you often add new and or or clauses as a whole line, that becomes much faster to add and remove those same lines as they're a single shortcut away in nearly all text editors.

Re: SQL Tips and Tricks

#83

Earlier quoted context omitted.

Instead of joining tables and using distinct or similar to filter rows, consiser using subquery "columns", ie in SELECT list. What does this mean? Running SELECT column1, ( SELECT column2, column3, ... FROM table_b WHERE table_a.id = table_b.a_id ) FROM table_a Results in "subquery must return only one column" as I expected. You mean returning the multiple columns as a record / composite type? Keep in mind GROUP BY c…

Sorry, was on mobile so hadn't patience to type examples. SELECT column1, ( SELECT column2 FROM table_b WHERE table_a.id = table_b.a_id ) as b_column2, ( SELECT column3 FROM table_b WHERE table_a.id = table_b.a_id ) as b_column3 FROM table_a It might look like a lot more work, but in my experience it's usually a lot faster. YMMV but check it.

How well that performs compared to a JOIN can vary massively depending on the data sizes of table_a & tale_b, how table_b is indexed, and what else is going on in the query.

If table_b has an index on id,column2,column3 (or on id INLUDEing column2,column3) I would expect the equivalent JOIN to usually be faster. If you have a clustered index on Id (which is the case more often than not in MS SQL Server and MySQL/InnoDB) then that would count for this unless the table is much wider than those three columns (so the index with its selective data would get many rows per page more than the base data).

Worst (and fairly common) case with sub-selects like that is the query planner deciding to run each subquery one per row from table_a. This is not an issue if you are only returning a few rows, or just one, from table_a, but in more complex examples (perhaps if this fragment is a CTE or view that is joined in a non-sargable manner so filtering predicates can't push down) you might find a lot more rows are processed this way even if few are eventually returned due to other filters.

There are times when the method is definitely faster but be very careful with it (test with realistic data sizes and patterns) because often when it isn't, it really isn't.

Re: SQL Tips and Tricks

#84

Earlier quoted context omitted.

A lot of databases support trailing commas in select clauses. Which is just as well. I want to scratch my eyes out every time I see someone formatting with comma starting the lines. It's the kind of foolish consistency that is a big part of performative engineering.

> A lot of databases support trailing commas in select clauses. Which ones? Postgres, Oracle, SQL Server, MySQL, MariaDB and SQLite do not allow that.

I guess I'm being spoiled by BigQuery :)

To be fair, BigQuery SQL is improving at quite a pace. If you follow their RSS, they are often announcing small but solid affordances like trailing commas, the new RANGE datatype, BigLake, some limited grouping and equality for arrays and structs, etc.

It is also probable that they expose Google's new query pipe syntax. Currently there are some hints from the error messages in the console that it's behind a feature flag or something.

Re: SQL Tips and Tricks

#85
post #19

Never use WHERE 1=1. It is both a security risk and a performance risk to run dynamic ad-hoc queries.

What is a dynamic, adhoc query? Why does adding 1=1 support that?

In this case. A query that you build by adding different strings. 1=1 is for adding AND statements to the WHERE clause dynamically. In your code. I never seen it used for anything else. Adhoc is just the practice of running raw SQL queries.

So you end up with things like this.

"SELECT * FROM Music WHERE 1=1" + "AND category='rock'"

The risk is now that you by mistake allow for SQL-injections but also every genre will generate a different query plan. Depending on what SQL engine you use this may hurt performance.

And one would think that this is a thing of the past. But it is not.

Re: SQL Tips and Tricks

#86
On readability, I often find aligning things in two columns is more readable. To modify the two examples in TFA:

    SELECT e.employee_id
         , e.employee_name
         , e.job
         , e.salary
      FROM employees e
     WHERE 1=1 -- Dummy value.
       AND e.job IN ('Clerk', 'Manager')
       AND e.dept_no != 5
         ;
and with a JOIN:

    SELECT e.employee_id
         , e.employee_name
         , e.job
         , e.salary
         , d.name
         , d.location
      FROM employees e
      JOIN departments d
           ON d.dept_no = e.dept_no
     WHERE 1=1 -- Dummy value.
       AND e.job IN ('Clerk', 'Manager')
       AND e.dept_no != 5
         ;
In the join example, for a simple ON clause like that I'll usually just have JOIN ... ON in the one line, but if there are multiple conditions they are usually clearer on separate lines IMO.

In more complicated queries I might further indent the joins too, like:

    SELECT *
      FROM employees e
           JOIN departments d
             ON d.dept_no = e.dept_no
     WHERE 1=1 -- Dummy value.
       AND e.job IN ('Clerk', 'Manager')
       AND e.dept_no != 5
         ;
YMMV. Some people strongly agree with me here, others vehemently hate the way I align such code…

WRT “Always specify which column belongs to which table”: this is particularly important for correlated sub-queries, because if you put the wrong column name in and it happens to match a name in an object in the outer query you have a potentially hard to find error. Also, if the table in the inner query is updated to include a column of the same name as the one you are filtering on in the outer, the meaning of your sub-query suddenly changes quite drastically without it having changed itself.

A few other things off the top of my head:

1. Remember that as well as UNION [ALL], EXCEPT and INTERSECT exist. I've seen (and even written myself) some horrendous SQL that badly implements these behaviours. TFA covers EXCEPT, but I find people who know about that don't always know about INTERSECT. It is rarely useful IME, but when it is useful it is really useful.

2. UPDATEs that change nothing still do everything else: create entries in your transaction log (could be an issue if using log-shipping for backups or read-only replicas etc.), fire triggers, create history rows if using system-versioned tables, and so forth. UPDATE a_table SET a_column = 'a value' WHERE a_column 'a value' can be a lot faster than without the WHERE.

3. Though of course be very careful with NULLable columns and/or setting a value NULL with point 2. “WHERE a_column IS DISTINCT FROM 'a value'” is much more maintainable if your DB supports that syntax (added in MS SQL Server 2022 and Azure SQL DB a little earlier, supported by Postgres years before, I don't know about other DBs without checking) than the more verbose alternatives.

4. Trying to force the sort order of NULLs with something like “ORDER BY ISNULL(a_column, 0)”, or doing similar with GROUP BY, can be very inefficient in some cases. If you expect few rows to be returned and there are relatively few NULLs in the sort target column it can be more performant to SELECT the non-NULL case and the NULL case then UNION ALL the two and then sort. Though if you do expect many rows this can backfire badly and you and up with excess spooling to disk, so test, test, and test again, when hacking around like this.

Re: SQL Tips and Tricks

#87

Earlier quoted context omitted.

Sorry, was on mobile so hadn't patience to type examples. SELECT column1, ( SELECT column2 FROM table_b WHERE table_a.id = table_b.a_id ) as b_column2, ( SELECT column3 FROM table_b WHERE table_a.id = table_b.a_id ) as b_column3 FROM table_a It might look like a lot more work, but in my experience it's usually a lot faster. YMMV but check it.

How well that performs compared to a JOIN can vary massively depending on the data sizes of table_a & tale_b, how table_b is indexed, and what else is going on in the query. If table_b has an index on id,column2,column3 (or on id INLUDEing column2,column3) I would expect the equivalent JOIN to usually be faster. If you have a clustered index on Id (which is the case more often than not in MS SQL Server and MySQL/Inno…

> perhaps if this fragment is a CTE or view

Yeah I guess I should have specified that this technique usually works best when done in the outer query, not buried deep inside.

It can be particularly effective if you fetch partial results, ie due to pagination or similar.

That said, these things aren't set in stone. I shared my experience, but my first tip goes first :)

Re: SQL Tips and Tricks

#88

The "readability" section has 3 examples. The first 2 are literally sacrificing readability so it's easier to write, and the last has an unreadable abomination that indenting is really not doing much.

Why do you think its worse

? I don’t see any problems

, or anything wrong with it

.

Re: SQL Tips and Tricks

#89
post #19

Earlier quoted context omitted.

What is a dynamic, adhoc query? Why does adding 1=1 support that?

Lets say its 2001 and you are writing some hot e-commerce stuff in plain php. You want to filter data depending on multiple fields in the submitted form. If some field is there, you add one more "AND" clause to the "WHERE", like this: if (isset($_POST['product'])) { $query .= "AND product = " . $_POST['product']; }. So in order not to check every time if the added clause is the first one you start with "WHERE 1=1 ",…

Php has nothing like this?

In [1]: "... WHERE " + " AND ".join(str(i) for i in range(4))

Out[1]: '... WHERE 0 AND 1 AND 2 AND 3'

Very strange.

Re: SQL Tips and Tricks

#90

Earlier quoted context omitted.

What about `WHERE true`?

I don't understand the point at all. If you need to add some condition later on, why not just add it then? What benefit is there to just marking out the spot where you might add the condition at some point in the future?

For their one here it's just the ability to rapidly comment/uncomment conditions in a query editor while exploring the data or debugging the query, and not having to worry about the leading AND or OR.

I've also seen it in code with iterative adds, for example:

  for crit in criteria:
      sql += " AND " + crit
No needed to add a sentinel or other logic to skip the first AND. I saw it a lot before people got used to " AND ".join(criteria).
Post reply on HN