Earlier quoted context omitted.
Proprietary device drivers come to mind..
They don't tend to be huge && in C
AMD's linux graphics driver has entered the chat
Important PostgreSQL 14 update to avoid silent corruption of indexes
81–90 of 101 posts
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#82It is noted in the post, I'll repeat it to be clear: This corruption can only occur _during_ a (re)index with CONCURRENTLY specified, on rows that are modified during the reindex operation, and only for that index. No other indexes are impacted, and an index can only be impacted when the updates on the table don't update indexed columns. Nevertheless, if you frequently run CIC, you could be having this issue -- right…
Simpler fix is to upgrade to fixed version and reindex concurrently any indexes that could have been affected.
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#83Earlier quoted context omitted.
Seems like something the compiler should take care of.
It can when you use LTO, but that tends to be very slow for large programs.
It is slow. Our (C++) project's MSVC release build ends with a glorious 2-minute run of link.exe with lto (/LTCG /O2) and aggressive inlining (/Ob3) enabled.
Limiting unit size in C++ helps with faster edit/compile/run cycles as well, which doesn't seem to be concern for C codebases in 21st century.
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#84It is noted in the post, I'll repeat it to be clear: This corruption can only occur _during_ a (re)index with CONCURRENTLY specified, on rows that are modified during the reindex operation, and only for that index. No other indexes are impacted, and an index can only be impacted when the updates on the table don't update indexed columns. Nevertheless, if you frequently run CIC, you could be having this issue -- right…
Simpler fix is to upgrade to fixed version and reindex concurrently any indexes that could have been affected.
That fixed version is not yet released (that is planned for 2022-06-16, in 6 days), and database upgrades are not trivial.
Sure, it's easier to "just go to the version that has this fixed", but if that's not possible, this might just be the next best thing.
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#85Earlier quoted context omitted.
> Everyone seems to be downplaying this corruption issue saying it only happens when x or y and it can be avoided by doing complex operation z but… I don't say corruption is not happening, but I clarify that you might not be affected: this corruption is not as "silent" as the title of the original post would make you think (it only occurs during concurrent reindex/index creation, which is not something that occurs na…
this corruption is not as "silent" Whether corruption is silent or not has no relation to how it can be caused. It just means that the corruption is not automatically detected and/or resolved, and the system will happily use the corrupted data. Which seems to be the case here, as you admit that both detection and fixing require manual intervention.
The title implies that data is silently being corrupted, which is true, but in well-known and detectable workloads (CIC/RIC).
However, contrary to what the title would make you think, once they are created the indexes do not get more corrupt over time; the corruption does not spread. Instead, they get less and less corrupt over time as the rows missing from the index are either updated in a non-HOT manner, or deleted.
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#86pg_amcheck can be helpful for validation if its a b-tree index
FWIW, amcheck can be useful but I am not 100% sure that it can detect all the broken cases either.
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#87It’s impressive that this is a single revert. That speaks to how the development of Postgres is done atomically. Also not surprised to see it was the EDB team with the expertise to fix it. Their model is a little outdated but they have a lot of experts working there.
I'm not sure who you're referring to as the EDB team. Please share what make you think that. The commit [1] (dug up by someone else in this discussion), references 4 people (including the committer), and only one of them seems to be EDB employee. (Info gleaned from their respective LinkedIn profiles, and from email signatures sent to pgsql-hackers list) [1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commi…
All of those mentioned in the commit helped in the diagnosis, while Andres was the one to detail the full chain of events that describes how the corruption happened, in [0].
[0] https://www.postgresql.org/message-id/20220524190133.j6ee7zh...
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#88Earlier quoted context omitted.
yes are people using editors that don't let them have multiple views of the same file or something?
A filepath is an index and a hiearchy that adds information and structure. It can't be completely replaced by editor affordances.
Depends on the language. C# somewhat replaces paths with namespaces, then you navigate classes and methods with editor tooling. I remember back when I was on Visual Studio writing C++ that they did something similar.
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#89Earlier quoted context omitted.
I worked on Oracle and MySQL for ages and seen tons of bugs. As a support company, the number of bugs we encounter with Postgres are none or hardly 1 in an year. I am still curios to see what those bugs are that made one scary !!!
Yeah, ORA-00600 comes to mind. At my previous company we had to restore from backups once because of this error.
ORA-600 just means an Oracle process crashed while executing a query. Could be a bug in the query optimizer or a dead disk controller. Just restoring a backup without understanding the root cause seems like setting up the next occurrence...
Re: Important PostgreSQL 14 update to avoid silent corruption of indexes
#90It is noted in the post, I'll repeat it to be clear: This corruption can only occur _during_ a (re)index with CONCURRENTLY specified, on rows that are modified during the reindex operation, and only for that index. No other indexes are impacted, and an index can only be impacted when the updates on the table don't update indexed columns. Nevertheless, if you frequently run CIC, you could be having this issue -- right…
What? That seems absurdly apologist even for me and I love postgres. Everyone seems to be downplaying this corruption issue saying it only happens when x or y and it can be avoided by doing complex operation z but… if build anything that isn’t a toy or demo on Postgres you NEED to build indexes concurrently. They shipped optimization to a feature to let indexes be built concurrently yet failed to perform thorough eno…
There's nothing apologist about the comment you're replying to. The parent is not defending the behaviour; they are explaining which scenarios are affected, and what the appropriate mitigations are. Nobody here is denying that it's a serious issue.