Live data from Hacker News

Results of the SQL Performance Quiz

use-the-index-luke.com

91–98 of 98 posts

Re: Results of the SQL Performance Quiz

#91

Wow this just highlights why I absolutely hate working with raw SQL be it Postgres or MSSQL.

SQL is extremely powerful. Whenever something comes up and says that it's going to hide the complexity of SQL/raw DB access, it's lying. :) I can work well in many cases, then it bites you in the ass later. That's not a reason to not use ORMs for example, but without knowing how the underlying DB works you can have problems. Abstractions are leaky, always. Same applies if you work in a high level language and you don…

Indeed. I would actually go further and say it can be a reason not to use ORMs though. But the rest of this is to bolster your point and hopefully give you some more ammo for your case.

One of the reasons I started working on the PGObject Perl framework[1] was because I found that if I hand-coded the SQL and programmed around that, I was more productive, and had fewer bugs.

The approach taken (based on our experience with LedgerSMB) is to focus on rules for mapping object oriented calls to stored procedure calls. The framework is of course PostgreSQL-only (and relies a lot on system catalog lookups).

My experience, backing what you are saying, is that if you know what you are doing, and you write good, maintainable stored procedures/UDF's then the power is totally worth it. Unfortunately, there are still some obstacles, and most of these are tooling.

I think the hatred of SQL from the parent (in the context of this quiz) is that if you separate performance from result, as SQL does, then you have to think about performance differently. In theory you code for results first and then tune performance later. In practice, this only gets you so far and you need to code with some knowledge of performance limitations. I know I have been bitten as badly as anyone else.

But, the tradeoff is that when I write more of my code into my queries, I get better performance, fewer bugs, and a significantly smaller codebase. The knowledge is worth it warts and all.

(When I have some really nice code samples, I will submit to HN)

[1] The official repos are the repos with names starting with "PGObject" at https://github.com/ledgersmb

Re: Results of the SQL Performance Quiz

#92

Earlier quoted context omitted.

Ugly compared to what query language of equal power and expressiveness?

Datalog. I honestly love SQL...I consider it a noble language:) It is the only declarative language out there that is still used today, and there are some solid reasons for that. However, I believe the biggest mistake that was made in the creation of SQL was to try to emulate natural language flow patterns. This makes it hard to format and edit, and when people optimize towards writability, they kill readability (suc…

The biggest mistake in SQL was in using NULLs to mean three different things (no record found in an outer join, vs not applicable, vs not yet known). Fortunately with some types (varchar) on sane db's you can use dedicated empty values ('' for varchar) for not applicable, but you are still stuck with the other two as possibly conflicting.

This is actually a big issue because your ability to have complex declarative constraints goes up with table width, so breaking off potentially nullable columns also removes them from being available for cross-column check constraints.

Re: Results of the SQL Performance Quiz

#93
post #5

Regarding #5: >That caught me by surprise. Both options “roughly the same” and “depends on the data” got about 25% — the guessing probably. I don't think it was guessing so much as reasoning that fetching 100 rows (and filtering by value) instead of 10 rows doesn't have significant real-world impact unless the row data is particularly large. I'll admit I didn't think of the out-of-index lookup, but my main thought wa…

Yep, that's exactly what I felt too. The only question I got wrong in fact.

Re: Results of the SQL Performance Quiz

#94
post #48

Earlier quoted context omitted.

Ugly compared to what query language of equal power and expressiveness?

Expressiveness? SQL is quite verbose for what it does achieve and if you look at an SQL with some JOINs, I'm not sure you want to uphold that opinion of its expressiveness. Don't get me wrong, I have no problem with SQL as such but it has some shortcomings. Probably most things can be blamed on its age. Why is the syntax of INSERT and UPDATE statements unnecessarily distinct? They do almost the same thing. Why are th…

So, yeah, but, like UPDATE can employ a WHERE clause, to apply conditional logic, whereas INSERT, not so much. You insert one record to a table. As long as you know the table? Boom. Done. Oh, but now you'd like to change some records in the table? Well, that's great, but which ones? Tell me where?

I think the whole readable natural language premise is a neat idea, but much in the same way that natural language can be easily complicated, so too, with SQL. Heap abstractions of logic and arithmetic on top, and it gets even worse. But it's still a cool idea. If it works out in your favor, you'll get self documenting code. But alas, there is no floor or cieling to the complexity one can introduce, be it deliberately or accidentally. And not only is it the query itself (or the author) that might over complicate matters. The data schema, defined by the DDL, can easily induce unwanted complexity in subtle ways, and this might be a problem deliberately placed beyond the reach of the person attempting to read or change the data locked within.

As for complaints about parsers, that's not a deficiency of the language. It's a deficiency of the tools used. If the development environment isn't presenting problems to the developer in a convenient manner, then the utility sucks. Get a new one. Don't blame the text-mode clients though. They are just as bare-bones as a command line, and deliberately so. At a certain level, SQL can be regarded the same way a shell script might be regarded, in terms of possessing the convenience of (or lack thereof) a basic TTY command interpretter. But there's no real reason why it should be impossible to create a SQL script authoring tool, which provides syntax validation as convenient as any compiled language might employ, when communicating compilation errors to the developer.

Re: Results of the SQL Performance Quiz

#95
I like some of the examples they provide, however this is very platform dependent. Some advanced platforms have highly optimized query engines, to where even bad queries can be handled if they are run many times. Expressions can be reordered without the users knowledge and the results will be the same.

Re: Results of the SQL Performance Quiz

#96
post #32

Earlier quoted context omitted.

But if it wasn't for the GROUP BY, filtering 100 results of a million down to 10 results wouldn't change performance much even if you read every column of every row of those 100. The trick is the fact that the GROUP BY means that "It used to return 100 it now returns 10" is a red herring, it still has to read every row to make up those 10.

I don't understand what you mean "trick". SELECT date_column, count(*) FROM tbl WHERE a = @a AND b = @b GROUP BY date_column; The "AND b = @b" causes the sql engine to access data in the table instead of solely relying on the index. GROUP BY has 0 to do with it. If you changed the query to SELECT a, date_column FROM tbl WHERE a = @a and SELECT a, date_column FROM tbl WHERE a = @a AND b = @b The answer would be the sa…

No, it wouldn't.

If we're told that:

SELECT a, date_column FROM tbl WHERE a = @a

Returns 100 rows.

Then:

SELECT a, date_column FROM tbl WHERE a = @a AND b = @b

Will only have to scan column b over 100 rows.

Even without an index that will always be neglible, not compared to using the index to grab 100 rows from 10million but just compared to running a query and returning results at all.

The reason that the original can be a lot slower is that the 100 and 10 rows of results are comprised of a lot more rows of actual information, because of the grouping.

You're right that:

SELECT a, date_column FROM tbl WHERE a = @a AND b = @b

would be a lot slower, given the same data, but that isn't the scenario, the group by has implications about what "returns 100 rows, returns 10 rows" actually means in terms of data read.

Re: Results of the SQL Performance Quiz

#97
post #96

Earlier quoted context omitted.

I don't understand what you mean "trick". SELECT date_column, count(*) FROM tbl WHERE a = @a AND b = @b GROUP BY date_column; The "AND b = @b" causes the sql engine to access data in the table instead of solely relying on the index. GROUP BY has 0 to do with it. If you changed the query to SELECT a, date_column FROM tbl WHERE a = @a and SELECT a, date_column FROM tbl WHERE a = @a AND b = @b The answer would be the sa…

No, it wouldn't. If we're told that: SELECT a, date_column FROM tbl WHERE a = @a Returns 100 rows. Then: SELECT a, date_column FROM tbl WHERE a = @a AND b = @b Will only have to scan column b over 100 rows. Even without an index that will always be neglible, not compared to using the index to grab 100 rows from 10million but just compared to running a query and returning results at all. The reason that the original c…

Query 1 is an index seek only. It does not access the table data.

Query 2 will perform the same index seek but will need to do a key lookup on each row and filter.

It's not negligible. The 100 results are not comprised of a lot more information in this case, regardless of the grouping, because the 1st query does not access the table.

Edit:

I happen to have a table laying around with a little over a million rows and set up a similar set of queries.

The query optimizer suggested the index seek taking 6% of total operation time while the key lookup taking up the other 94%. The rest was negligible.

Post reply on HN