Live data from Hacker News

Speedup of deletes on PostgreSQL

ivdl.co.za

41–50 of 64 posts

Re: Speedup of deletes on PostgreSQL

#41

Find missing indexes, return SQL to create them. SELECT CONCAT('CREATE INDEX ', relname, '_', conname, '_ix ON ', nspname, '.', relname, ' ', regexp_replace( regexp_replace(pg_get_constraintdef(pg_constraint.oid, true), ' REFERENCES.*$','',''), 'FOREIGN KEY ','',''), ';') AS query FROM pg_constraint JOIN pg_class ON (conrelid = pg_class.oid) JOIN pg_namespace ON (relnamespace = pg_namespace.oid) WHERE contype = 'f' A…

This seems to have a few false positives. Multi-column indexes that have the first column as the foreign key work as well as a single column index. For more modern postgres, partial indexes with `WHERE column_name IS NOT NULL` on columns that can be null are also valid and more performant.

Here's what we use in CI to check for missing indexes:

    -- Unindexed FK -- Missing indexes - For CI

    WITH y AS (
    SELECT
    pg_catalog.format('%I', c1.relname)  AS referencing_tbl,
    pg_catalog.quote_ident(a1.attname) AS referencing_column,
    (SELECT pg_get_expr(indpred, indrelid) FROM pg_catalog.pg_index WHERE indrelid = t.conrelid AND indkey[0] = t.conkey[1] AND indpred IS NOT NULL LIMIT 1) partial_statement
    FROM pg_catalog.pg_constraint t
    JOIN pg_catalog.pg_attribute  a1 ON a1.attrelid = t.conrelid AND a1.attnum = t.conkey[1]
    JOIN pg_catalog.pg_class      c1 ON c1.oid = t.conrelid
    JOIN pg_catalog.pg_namespace  n1 ON n1.oid = c1.relnamespace
    JOIN pg_catalog.pg_class      c2 ON c2.oid = t.confrelid
    JOIN pg_catalog.pg_namespace  n2 ON n2.oid = c2.relnamespace
    JOIN pg_catalog.pg_attribute  a2 ON a2.attrelid = t.confrelid AND a2.attnum = t.confkey[1]
    WHERE t.contype = 'f'
    AND NOT EXISTS (
    SELECT 1
    FROM pg_catalog.pg_index i
    WHERE i.indrelid = t.conrelid
    AND i.indkey[0] = t.conkey[1]
    AND indpred IS NULL
    )
    )
    SELECT  referencing_tbl || '.' || referencing_column as column
    FROM y
    WHERE (partial_statement IS NULL OR partial_statement  ('(' || referencing_column || ' IS NOT NULL)'))
    ORDER BY 1;


Additionally I have this to specify the index creation commands (CONCURRENTLY is recommended for existing tables in production as it doesn't cause locking):

    -- Unindexed FK -- Missing indexes - Show Create Syntax

    WITH y AS (
        SELECT
            pg_catalog.format('%I.%I', n1.nspname, c1.relname)  AS referencing_tbl,
            pg_catalog.quote_ident(a1.attname) AS referencing_column,
            (SELECT pg_get_expr(indpred, indrelid) FROM pg_catalog.pg_index WHERE indrelid = t.conrelid AND indkey[0] = t.conkey[1] AND indpred IS NOT NULL LIMIT 1) partial_statement,
            t1.typname AS referencing_type,
            t.conname AS existing_fk_on_referencing_tbl,
            pg_catalog.format('%I.%I', n2.nspname, c2.relname) AS referenced_tbl,
            pg_catalog.quote_ident(a2.attname) AS referenced_column,
            t2.typname AS referenced_type,
            pg_relation_size( pg_catalog.format('%I.%I', n1.nspname, c1.relname) ) AS referencing_tbl_bytes,
            pg_relation_size( pg_catalog.format('%I.%I', n2.nspname, c2.relname) ) AS referenced_tbl_bytes,
            pg_catalog.format($$CREATE INDEX CONCURRENTLY IF NOT EXISTS %I ON %s%I(%I)%s;$$, c1.relname || '_' || a1.attname || 'x' , CASE WHEN n1.nspname = 'public' THEN '' ELSE n1.nspname || '.' END, c1.relname, a1.attname, CASE WHEN a1.attnotnull THEN '' ELSE ' WHERE ' || a1.attname || ' IS NOT NULL' END) AS suggestion
        FROM pg_catalog.pg_constraint t
        JOIN pg_catalog.pg_attribute  a1 ON a1.attrelid = t.conrelid AND a1.attnum = t.conkey[1]
        JOIN pg_catalog.pg_type       t1 ON a1.atttypid = t1.oid
        JOIN pg_catalog.pg_class      c1 ON c1.oid = t.conrelid
        JOIN pg_catalog.pg_namespace  n1 ON n1.oid = c1.relnamespace
        JOIN pg_catalog.pg_class      c2 ON c2.oid = t.confrelid
        JOIN pg_catalog.pg_namespace  n2 ON n2.oid = c2.relnamespace
        JOIN pg_catalog.pg_attribute  a2 ON a2.attrelid = t.confrelid AND a2.attnum = t.confkey[1]
        JOIN pg_catalog.pg_type       t2 ON a2.atttypid = t2.oid
        WHERE t.contype = 'f'
        AND NOT EXISTS (
            SELECT 1
            FROM pg_catalog.pg_index i
            WHERE i.indrelid = t.conrelid
            AND i.indkey[0] = t.conkey[1]
            AND i.indpred IS NULL
        )
    )
    SELECT  referencing_tbl,
            referencing_column,
            existing_fk_on_referencing_tbl,
            referenced_tbl,
            referenced_column,
            pg_size_pretty(referencing_tbl_bytes) AS referencing_tbl_size,
            pg_size_pretty(referenced_tbl_bytes) AS referenced_tbl_size,
            suggestion
    FROM y
    WHERE (partial_statement IS NULL OR partial_statement  ('(' || referencing_column || ' IS NOT NULL)'))
    ORDER BY
        referencing_tbl_bytes DESC,
        referenced_tbl_bytes DESC,
        referencing_tbl,
        referenced_tbl,
        referencing_column,
        referenced_column;

Re: Speedup of deletes on PostgreSQL

#42
post #35

This is DBA 101 stuff. If a database is part of your software, you really need someone on the team who knows how it works.

You're not wrong, but unfortunately many teams don't. Probably my favourite "ya'll don't understand how databases work" was where they "reserved" space for MySQL enums; for example for the "active" column it would be something like: enum( 'active', 'deleted', '_futureval1', '_futureval2', '_futureval3', '_futureval4', '_futureval5', '_futureval6', '_futureval7', '_futureval8', '_futureval9' ) Enums don't work like th…

I'm not sure what the current state of things are since I haven't use MySQL recently but this used to be a perfectly valid thing to do.

The issue was that MySQL doesn't use a full int to store enums. If your enum has 8 values, it stores in 1 byte, if it has more than 8, it stores it in 2 bytes. Adding that 9th value thus requires re-writing the entire table. So yes - it can make sense to "reserve space" to avoid a future table re-write.

You also had to be careful to include `ALGORITHM=INPLACE, LOCK=NONE;` in your `ALTER TABLE` statement when changing the enum or it would lock the table and rewrite it.

Re: Speedup of deletes on PostgreSQL

#43
post #30
post #17

Earlier quoted context omitted.

I'll have to defend your parent commenter on this one. Not having indexes for FKs is on average much worse for overall performance. Defaults should be reasonable. In the great majority of cases you WANT to have indexes in FKs. > expect the universe to magically fix all of your mistakes This kind of derogatory hyperbole is not necessary nor productive. I should expect tools to help me avoid mistakes. Not having an ind…

I'll play devil's advocate. To be clear I generally agree that foreign keys should essentially always have a corresponding index, and that not including an index is a mistake far more often than it isn't. My only counterargument is that—especially in production—adding indexes is expensive. Adding foreign keys is cheap. Latching a potentially expensive operation that can result in downtime to what should be (and often…

Correct me if I'm wrong but, FKs are rarely created for existing columns.

You usually create the column and the FK in the same script. And usually starting with a NULL value for existing rows.

And if it's a new table then there's no rows anyway.

So the most common operations when creating FK's aren't expensive as far as I know.

You know what's expensive? Creating an index on a large table because you or your RDMS forgot to create the index when the FK was created and now JOINS are crawling to halt.

Re: Speedup of deletes on PostgreSQL

#44
post #42
post #35

Earlier quoted context omitted.

You're not wrong, but unfortunately many teams don't. Probably my favourite "ya'll don't understand how databases work" was where they "reserved" space for MySQL enums; for example for the "active" column it would be something like: enum( 'active', 'deleted', '_futureval1', '_futureval2', '_futureval3', '_futureval4', '_futureval5', '_futureval6', '_futureval7', '_futureval8', '_futureval9' ) Enums don't work like th…

I'm not sure what the current state of things are since I haven't use MySQL recently but this used to be a perfectly valid thing to do. The issue was that MySQL doesn't use a full int to store enums. If your enum has 8 values, it stores in 1 byte, if it has more than 8, it stores it in 2 bytes. Adding that 9th value thus requires re-writing the entire table. So yes - it can make sense to "reserve space" to avoid a fu…

A byte would fit at least 255 different values, right? How often is this limit exceeded in practice.

Re: Speedup of deletes on PostgreSQL

#45
post #31

What popular SQL databases need is an option/hint to return an error instead of taking a slow query plan. That way a lot of SQL index creation -- something considered a black art by surprisingly many -- would just be prompted by test suite failures. If you don't have the right indices, your test fails. Simple. In this case, have TestDeleteCustomer fail, realize you need to add index, 5 minutes later done and learned…

This seems like a very easy thing for any sort of middleware (or ORM) to do for you. Maybe even add typed where clauses that only exist for indexed columns.

Re: Speedup of deletes on PostgreSQL

#46
post #31

What popular SQL databases need is an option/hint to return an error instead of taking a slow query plan. That way a lot of SQL index creation -- something considered a black art by surprisingly many -- would just be prompted by test suite failures. If you don't have the right indices, your test fails. Simple. In this case, have TestDeleteCustomer fail, realize you need to add index, 5 minutes later done and learned…

This seems like a very easy thing for any sort of middleware (or ORM) to do for you. Maybe even add typed where clauses that only exist for indexed columns.

I don't understand the idea ... do you mean a middleware that parse the SQL query and contains a query planner and has full knowledge of all the indices in the database .. or something else?

Re: Speedup of deletes on PostgreSQL

#47
post #42
post #35

Earlier quoted context omitted.

You're not wrong, but unfortunately many teams don't. Probably my favourite "ya'll don't understand how databases work" was where they "reserved" space for MySQL enums; for example for the "active" column it would be something like: enum( 'active', 'deleted', '_futureval1', '_futureval2', '_futureval3', '_futureval4', '_futureval5', '_futureval6', '_futureval7', '_futureval8', '_futureval9' ) Enums don't work like th…

I'm not sure what the current state of things are since I haven't use MySQL recently but this used to be a perfectly valid thing to do. The issue was that MySQL doesn't use a full int to store enums. If your enum has 8 values, it stores in 1 byte, if it has more than 8, it stores it in 2 bytes. Adding that 9th value thus requires re-writing the entire table. So yes - it can make sense to "reserve space" to avoid a fu…

You can store 255 values in one byte, and reserving two bytes is not what that did.

And even if I did, it still leaves the inability to actually rename enums without scanning the full table at least twice (which still doesn't seem possible in MariaDB, unless I missed something there).

If you potentially want great flexibility you shouldn't be using enums in the first place but int and a relational mapping to another table.

Re: Speedup of deletes on PostgreSQL

#48
post #40
post #36

Earlier quoted context omitted.

Analyze collects statistics the query planner uses to determine the query plan. It can change the resulting plan, yes. Production databases using different query plans sure is annoying and cause problems, but I'm not so sure whether returning errors is better. "Slow" beats "not working at all" in almost all cases. The typical case it will select a different query plan once the data grows, which is not so straight-for…

Note that I am ONLY talking about a mode to use for limited, trivial OLTP style queries. The kind where the query planner will never be in doubt -- if you just have the right indices in place. The kind of simple backend software queries where people consider NoSQL instead to avoid SQL's oddities. The mode I talk about is very inappropriate for any kind of reporting or analytics query or ad hoc queries etc. > "Slow" b…

You don't know what the "right indexes" are, because sometimes "no index" is the "right index". Sometimes because a full table scan is faster. Sometimes because you're okay accepting the various performance trade-offs (e.g. insert speed vs. update speed, storage space on disk).

Many applications don't have tests for every single last trivial SQL query, and adding those just because the SQL server may decide to bail out because it might perhaps possibly could be 100ms slower is not a good way for most teams to spend their time.

In the end it's just trading one confusion for another confusion. But the current confusion has a lot less complexity overall, so that clearly the "better" one IMHO.

Re: Speedup of deletes on PostgreSQL

#49
post #39
post #16

Lacking indexes on columns involved in a foreign key will also cause deadlocks in Oracle. This problem is common. "Obviously, Oracle considers deadlocks a self-induced error on part of the application and, for the most part, they are correct. Unlike in many other RDBMSs, deadlocks are so rare in Oracle they can be considered almost non-existent. Typically, you must come up with artificial conditions to get one. "The…

Why aren’t indexes for FK relationships the default? If you really don’t want one there should be a hint/pragma to turn it off. It’s just such a stupid reason for a full table scan.

On Postgres, they are. You'll get an informational message saying the index was automatically generated.

But this is not normal behavior. I think Postgres is the only one that does this.

Re: Speedup of deletes on PostgreSQL

#50
post #43
post #30

Earlier quoted context omitted.

I'll play devil's advocate. To be clear I generally agree that foreign keys should essentially always have a corresponding index, and that not including an index is a mistake far more often than it isn't. My only counterargument is that—especially in production—adding indexes is expensive. Adding foreign keys is cheap. Latching a potentially expensive operation that can result in downtime to what should be (and often…

Correct me if I'm wrong but, FKs are rarely created for existing columns. You usually create the column and the FK in the same script. And usually starting with a NULL value for existing rows. And if it's a new table then there's no rows anyway. So the most common operations when creating FK's aren't expensive as far as I know. You know what's expensive? Creating an index on a large table because you or your RDMS for…

FK indexes by necessity need to be placed on the foreign table, which is just as likely to be a preexisting table that already contains data.

To be clear I 100% agree that adding indexes later is extremely painful. A little care when first creating tables goes a long way, and I’ve never seen a database fall over due to preemptive over-indexing but I’ve seen countless do so thanks to being underindexed.

Still, taking a DDL operation which is presumed to be essentially instantaneous and adding a default behavior that requires locking completely separate tables for a potentially-lengthy update does give me pause.

Post reply on HN