Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Important PostgreSQL 14 update to avoid silent corruption of indexes

migops.com

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#41

Post by colordrops »

throwawaylinux wrote: post #37
colordrops wrote: post #33

Earlier quoted context omitted.

The point of calling it a nitpick was to indicate precisely that my comment wasn't meant to be taken harshly.

But nitpick still means it is a nit, I don't really think it is at all. And you're saying it in response to someone who said the code is very clean, which is quite petty.

> nitpick: engage in fussy or pedantic fault-finding.

Yes, I'm fully aware that the comment is petty. The point is to communicate that I don't disagree that the code is clean and well written.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#42

Post by rattray »

snuxoll wrote: post #38
rattray wrote: post #30

Earlier quoted context omitted.

To be fair, I think most users of Postgres hold it to a higher standard than the software they write themselves, and reasonably so. Postgres has developed a sterling reputation for reliability, and this does tarnish it somewhat. (Not by much in my personal view, but not zero either).

Not the first time there’s been a PostgreSQL bug like this, like always it will get a regression test and the PGDG will be better for it. It’s not like other database products haven’t had very similar issues, either [ https://support.microsoft.com/en-us/topic/fix-data-corruptio... ] Hell, last night I literally had a MSSQL server where Windows Server Failover Clustering randomly decided to delete all of the AlwaysOn…

Sure.

But notice how we're now comparing Postgres to MSSQL? Generally I think of it as head-and-shoulders better than it. I still do, but by a little less. That's okay.

Did they act responsibly, fix it quickly, and communicate effectively? Yes, tremendously so – which makes me feel overall better about using Postgres generally. But, I'll feel ever so slightly more hesitant about adopting a major version shortly after it comes out next time. Again, that's okay.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#43

Post by fathyb »

bigiain wrote: post #39
SonOfLilit wrote: post #17

Earlier quoted context omitted.

Large C codebases _have_ to be exceptionally nice, or they immediately collapse under their own weight. As a dev team, the language teaches you this the hard way. I've never seen a terrible huge C codebase (but have seen many in other languages).

> I've never seen a terrible huge C codebase I have 100% confidence they exist. They just don't get uploaded to Github out of shame or embarrasment.

Proprietary device drivers come to mind..

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#44

Post by colordrops »

cozzyd wrote: post #34
jvolkman wrote: post #13

Earlier quoted context omitted.

I get the impression that long files are culturally acceptable in systems-level C code. E.g., just a cherry-picked file from Linux: kernel/sched/core.c is over 11k lines. https://github.com/torvalds/linux/blob/master/kernel/sched/c...

yes are people using editors that don't let them have multiple views of the same file or something?

A filepath is an index and a hiearchy that adds information and structure. It can't be completely replaced by editor affordances.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#45

Post by fjfbsufhdvfy »

colordrops wrote: post #31
gnulinux wrote: post #14

Earlier quoted context omitted.

Quite the opposite, I hate it when projects have dozens upon dozens of modules with 1 function. Multiple huge files are the best sweet spot. (Only crazy adn exceptional things like putting everything into a single file damages the readability imho)

Files with one function vs files with 5000 lines are not the only two options.

I've got it! We could use files with one function that spans 5000 lines.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#46

Post by fjfbsufhdvfy »

colordrops wrote: post #32
plq wrote: post #29

Earlier quoted context omitted.

While I'm not sure that was a consideration here, sometimes C compilers produce better machine code when they got access to more function definitions. Eg. Sqlite recommends that embedders use the single ~10mb sqlite.c file[1] for both ease of use and performance reasons. [1]: https://www.sqlite.org/amalgamation.html

Seems like something the compiler should take care of.

It can when you use LTO, but that tends to be very slow for large programs.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#47

Post by RedShift1 »

rattray wrote: post #30
claytongulick wrote: post #25

Earlier quoted context omitted.

Software has bugs. Sometimes they're missed in testing. Do you hold the software that you write to the same standard?

To be fair, I think most users of Postgres hold it to a higher standard than the software they write themselves, and reasonably so. Postgres has developed a sterling reputation for reliability, and this does tarnish it somewhat. (Not by much in my personal view, but not zero either).

So out of all the bugs you choose this one for tarnishing its reputation? Don't go looking for postgresql and fsync then...

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#48

Post by eurasiantiger »

throwawaydingus wrote: post #19
mattashii wrote: post #5

It is noted in the post, I'll repeat it to be clear: This corruption can only occur _during_ a (re)index with CONCURRENTLY specified, on rows that are modified during the reindex operation, and only for that index. No other indexes are impacted, and an index can only be impacted when the updates on the table don't update indexed columns. Nevertheless, if you frequently run CIC, you could be having this issue -- right…

What? That seems absurdly apologist even for me and I love postgres. Everyone seems to be downplaying this corruption issue saying it only happens when x or y and it can be avoided by doing complex operation z but… if build anything that isn’t a toy or demo on Postgres you NEED to build indexes concurrently. They shipped optimization to a feature to let indexes be built concurrently yet failed to perform thorough eno…

No post body was provided.

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#49

Post by wahern »

osigurdson wrote: post #28
gnulinux wrote: post #14

Earlier quoted context omitted.

Quite the opposite, I hate it when projects have dozens upon dozens of modules with 1 function. Multiple huge files are the best sweet spot. (Only crazy adn exceptional things like putting everything into a single file damages the readability imho)

300 to 1000 lines per file is best IMO.

IMO, paintings with the color blue are the best.

What does number of lines have to do with anything?

Re: Important PostgreSQL 14 update to avoid silent corruption of indexes

#50

Post by Underphil »

jtc331 wrote: post #24
TedShiller wrote: post #20

I love Postgres. But when I look at the bug list of every release it makes me scared. The types of bugs they have are indicative of a poor development process.

No, it's indicative of the complexity of the system. Please show other similarly complex systems with fewer bugs: basically doesn't happen.

Yep. We all have dirty laundry, it's just that we can see it open-source projects.
Post reply on HN