Live data from Hacker News

Important PostgreSQL 14 update to avoid silent corruption of indexes

migops.com

51–60 of 101 posts

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#51

Earlier quoted context omitted.

This feels like a good area for tooling (editors, source hosts, SCM extensions) to improve experience. I don’t always mind large source files (and sometimes may prefer them over large file system hierarchies), but the can be a pain to navigate in some circumstances. As an example, making several related changes in very different parts of a file, where you need to cross-reference between them. The changes themselves m…

This is how older editors like emacs work. You interact with views/windows/tabs called buffers and those buffers can have files loaded into them. Multiple buffers can reference the same code file but view different sections simultaneously. So you can investigate or edit different parts of one huge file the same way you would smaller ones.

You just blew my mind.

I use JOE and JOE also supports multiple views into the same file. In fact, to open another file is two commands: the open view/window (^KO) command followed by edit file command (^KE). I've always used this facility for as long as I can remember and it never really occurred to me until now that people using more modern editors and especially GUI editors may not enjoy this same convenience--either not possible or no simple chain of command inputs to get there. And it's not like I don't use GUI editors, just not in situations where I would realize this feature was missing.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#52

It’s impressive that this is a single revert. That speaks to how the development of Postgres is done atomically. Also not surprised to see it was the EDB team with the expertise to fix it. Their model is a little outdated but they have a lot of experts working there.

I'm not sure who you're referring to as the EDB team. Please share what make you think that.

The commit [1] (dug up by someone else in this discussion), references 4 people (including the committer), and only one of them seems to be EDB employee. (Info gleaned from their respective LinkedIn profiles, and from email signatures sent to pgsql-hackers list)

[1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#53

Earlier quoted context omitted.

Actually I don’t mind big files. It is simpler scanning through it or doing a quick search than if you had a bunch of smaller files. And 5000 lines is not awkward for most editors, especially as many editors have the ability to collapse functions.

This feels like a good area for tooling (editors, source hosts, SCM extensions) to improve experience. I don’t always mind large source files (and sometimes may prefer them over large file system hierarchies), but the can be a pain to navigate in some circumstances. As an example, making several related changes in very different parts of a file, where you need to cross-reference between them. The changes themselves m…

I’m glad to see I’ve brought the emacs/vim people together for a common purpose!

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#54

Earlier quoted context omitted.

Files with one function vs files with 5000 lines are not the only two options.

I've got it! We could use files with one function that spans 5000 lines.

How about a function that spans 5000 files

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#55

It is noted in the post, I'll repeat it to be clear: This corruption can only occur _during_ a (re)index with CONCURRENTLY specified, on rows that are modified during the reindex operation, and only for that index. No other indexes are impacted, and an index can only be impacted when the updates on the table don't update indexed columns. Nevertheless, if you frequently run CIC, you could be having this issue -- right…

What? That seems absurdly apologist even for me and I love postgres. Everyone seems to be downplaying this corruption issue saying it only happens when x or y and it can be avoided by doing complex operation z but… if build anything that isn’t a toy or demo on Postgres you NEED to build indexes concurrently. They shipped optimization to a feature to let indexes be built concurrently yet failed to perform thorough eno…

Those type of issues happened with Oracle database, Microsoft sql server, MySQL.

That’s the reason why you should always be at least one major release behind the latest.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#56
post #51

Earlier quoted context omitted.

This is how older editors like emacs work. You interact with views/windows/tabs called buffers and those buffers can have files loaded into them. Multiple buffers can reference the same code file but view different sections simultaneously. So you can investigate or edit different parts of one huge file the same way you would smaller ones.

You just blew my mind. I use JOE and JOE also supports multiple views into the same file. In fact, to open another file is two commands: the open view/window (^KO) command followed by edit file command (^KE). I've always used this facility for as long as I can remember and it never really occurred to me until now that people using more modern editors and especially GUI editors may not enjoy this same convenience--eit…

You can do this in vscode as well, even have it side by side. I’m missing something?

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#57
post #24

Earlier quoted context omitted.

No, it's indicative of the complexity of the system. Please show other similarly complex systems with fewer bugs: basically doesn't happen.

Yep. We all have dirty laundry, it's just that we can see it open-source projects.

And at least, we see it. It's much easier to appreciate the threat when you see the submerged part of the iceberg :-)

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#58

Earlier quoted context omitted.

File could definitely be broken up a bit. Over 5000 lines! Just a nitpick though.

I get the impression that long files are culturally acceptable in systems-level C code. E.g., just a cherry-picked file from Linux: kernel/sched/core.c is over 11k lines. https://github.com/torvalds/linux/blob/master/kernel/sched/c...

There are 18609 .c files in my checked-out copy of the FreeBSD src tree. The median length is 258 lines; 90% are 1373 lines or shorter; 99% are 5241 lines or shorter.

The statistics for the 6071 .c files in the FreeBSD kernel are somewhat higher -- median is 460 lines; 90th percentile is 2070 lines; 99th percentile is 7678 lines -- but your example of a 11133 line file is definitely at the extreme high end.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#59

Earlier quoted context omitted.

I've got it! We could use files with one function that spans 5000 lines.

How about a function that spans 5000 files

Why even use 5000 lines when you you can often combine them all into one line.
Post reply on HN