Live data from Hacker News

Introduction to Window Functions in SQL

khashtamov.com

11–20 of 47 posts

Re: Introduction to Window Functions in SQL

#11
post #6

I've always wondered and perhaps someone here might know....do postgres' CTEs translate into big select/sub select queries under the hood? Or are they something special entirely? Ie (forgive formatting as I'm on phone) does: With mything as ( Select * from table where... ), Myotherthing as ( Select * from mything where... ) Get translated to Select * from (select * from ( select * from...)...)...) So I'm just wonderi…

It's a bit hidden (or rather not pronounced enough) in the general docs but there are performance implications of using CTEs:

https://www.postgresql.org/docs/13/queries-with.html

> A useful property of WITH queries is that they are normally evaluated only once per execution of the parent query, even if they are referred to more than once by the parent query or sibling WITH queries. Thus, expensive calculations that are needed in multiple places can be placed within a WITH query to avoid redundant work. Another possible application is to prevent unwanted multiple evaluations of functions with side-effects. However, the other side of this coin is that the optimizer is not able to push restrictions from the parent query down into a multiply-referenced WITH query, since that might affect all uses of the WITH query's output when it should affect only one. The multiply-referenced WITH query will be evaluated as written, without suppression of rows that the parent query might discard afterwards. (But, as mentioned above, evaluation might stop early if the reference(s) to the query demand only a limited number of rows.)

> However, if a WITH query is non-recursive and side-effect-free (that is, it is a SELECT containing no volatile functions) then it can be folded into the parent query, allowing joint optimization of the two query levels. By default, this happens if the parent query references the WITH query just once, but not if it references the WITH query more than once. You can override that decision by specifying MATERIALIZED to force separate calculation of the WITH query, or by specifying NOT MATERIALIZED to force it to be merged into the parent query. The latter choice risks duplicate computation of the WITH query, but it can still give a net savings if each usage of the WITH query needs only a small part of the WITH query's full output.

There are some examples below this text as well.

Re: Introduction to Window Functions in SQL

#12
I find window functions to be an excellent way to find the max version of a set of things. The trick is to partition by some columns (similar to how you would use a group by), order by descending on your version number field, and use the row_number() function which is very lightweight. Then you filter for all entries where rownumber = 1 and voila you have the max version without having to link back on yourself!

Re: Introduction to Window Functions in SQL

#13
post #6

I've always wondered and perhaps someone here might know....do postgres' CTEs translate into big select/sub select queries under the hood? Or are they something special entirely? Ie (forgive formatting as I'm on phone) does: With mything as ( Select * from table where... ), Myotherthing as ( Select * from mything where... ) Get translated to Select * from (select * from ( select * from...)...)...) So I'm just wonderi…

It's a bit hidden (or rather not pronounced enough) in the general docs but there are performance implications of using CTEs: https://www.postgresql.org/docs/13/queries-with.html > A useful property of WITH queries is that they are normally evaluated only once per execution of the parent query, even if they are referred to more than once by the parent query or sibling WITH queries. Thus, expensive calculations that a…

thanks very much for the reply, it was a case of RTFM on my end!

Re: Introduction to Window Functions in SQL

#14
post #9
post #6

I've always wondered and perhaps someone here might know....do postgres' CTEs translate into big select/sub select queries under the hood? Or are they something special entirely? Ie (forgive formatting as I'm on phone) does: With mything as ( Select * from table where... ), Myotherthing as ( Select * from mything where... ) Get translated to Select * from (select * from ( select * from...)...)...) So I'm just wonderi…

In theory, both forms should get optimized similarly by the DB, but the practice will likely differ from database to database, and maybe even from DB version to DB version. There are though things that CTEs can do and sub-selects can't (e.g. WITH RECURSIVE)

Didn't know about WITH RECURSIVE, very cool - thanks

Re: Introduction to Window Functions in SQL

#15
post #6

I've always wondered and perhaps someone here might know....do postgres' CTEs translate into big select/sub select queries under the hood? Or are they something special entirely? Ie (forgive formatting as I'm on phone) does: With mything as ( Select * from table where... ), Myotherthing as ( Select * from mything where... ) Get translated to Select * from (select * from ( select * from...)...)...) So I'm just wonderi…

Not 100% about the newest versions of Postgres. But certainly in older versions CTEs created query planner boundaries. So the planner would optimise each CTE separately, but wouldn’t optimise the entire query with all CTEs together, which of course can result in some slightly nonsensical query plans. To my knowledge this is considered a limitation rather than a feature as it can cause performance issues with some que…

In postgresql 12 a patch [0] is merged to remove the optimization fence certain conditions. From the changelog [1]

" Allow common table expressions (CTEs) to be inlined into the outer query (Andreas Karlsson, Andrew Gierth, David Fetter, Tom Lane)

Specifically, CTEs are automatically inlined if they have no side-effects, are not recursive, and are referenced only once in the query. Inlining can be prevented by specifying MATERIALIZED, or forced for multiply-referenced CTEs by specifying NOT MATERIALIZED. Previously, CTEs were never inlined and were always evaluated before the rest of the query. "

[0] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... [1] https://www.postgresql.org/docs/12/release-12.html

Re: Introduction to Window Functions in SQL

#16
I like how the article teaches by explicit example. Very helpful when it comes to data.

The "lag" and "lead" SQL functions are particularly useful when analysing sequences of timestamped events (GPS fixes, task completions etc). They allow you to easily return the delta of a value (time/distance) between the current and previous row, which is really useful if you want to compare expected vs actual on those deltas.

Re: Introduction to Window Functions in SQL

#17
A few cool tricks I use with window functions:

1- To find blocks of contiguous values, you can use something similar to Gauss' trick for calculating arithmetic progressions: sort them by descending order and add each value to the row number. All contiguous values will add to the same number. You can then apply max/min and get rows that correspond to the blocks of values.

    select min(n), max(n) from (
      select n, n+row_number() over (order by n desc) group
      from numbers
    )
    group by group
    order by 1
2- You can use a window function with exponential/logarithms in order to calculate the accumulated inflation for the last n months:

    select date, inflation, (exp(accumulated)-1)*100 from (
      select date, inflation, sum(ln(1+(inflation/100))) over (order by date desc rows between current row and 11 following) as accumulated
      from inflation
    ) 
3- You can do all the paging in SQL (fetch page n of m) or simply add a column with the total number of rows (this often makes it easier to process the results).

Re: Introduction to Window Functions in SQL

#18
Analytic functions as they are known by in Oracle, SQL have been around for more than a decade. They are useful for calculating running balances, getting a previous or next row value, doing a "group by" for a subset of columns.. etc. They're the best thing in SQL "since sliced bread".

https://towardsdatascience.com/analytical-functions-in-oracl...

Re: Introduction to Window Functions in SQL

#19
Question for the pros: In doing some data engineering work, I found that creating temporary tables and dropping them after the run was much more performant and memory-efficient than using CTEs. No other change was made to the queries in the CTE, just putting them in a separate CREATE TABLE AS... script before the part that needed the calculations.

Why is this the case? Shouldn't CTEs be more efficient?

Re: Introduction to Window Functions in SQL

#20
post #19

Question for the pros: In doing some data engineering work, I found that creating temporary tables and dropping them after the run was much more performant and memory-efficient than using CTEs. No other change was made to the queries in the CTE, just putting them in a separate CREATE TABLE AS... script before the part that needed the calculations. Why is this the case? Shouldn't CTEs be more efficient?

Until postgres 12 (if you were using postgres) CTEs were an optimization fence. Where filters would not get pushed down into the CTE if they were only specified outside the CTE (but in fields from the CTE).

https://paquier.xyz/postgresql-2/postgres-12-with-materializ...

Post reply on HN