[flagged]
Hacking on PostgreSQL is hard
21–30 of 56 posts
Re: Hacking on PostgreSQL is hard
#22lol, of course the first comment is essentially “it would probably be easier if it were written in Rust”.
Re: Hacking on PostgreSQL is hard
#23A few things to note
- Postgres documentation is one of the well maintained database documentations. This also means that developers, committers ensure changes to documentations for every relevant patch.
- talk about bugs in postgres compared to MySQl or Oracle or etc databases. Nugs are comparatively lesser or generally rare even if you are supporting postgres services as a vendor with lots of customer. the reason is the efforts involved by a strong team of developers in not accepting anything and everything, there are strict best practices, reviews, discussions, tests, and a lot more that makes it difficult to pass to a release.
- ultimately, more easy is the acceptance of a patch, more the number of bugs.
I love Postgres the way it is today and it still is the dbms of the year and developers most loved database.
I wish we have more Contributors committers, developers and also users and companies supporting Postgres so that the time to push a feature gets more faster and reasonable easier with more support.
Re: Hacking on PostgreSQL is hard
#24lol, of course the first comment is essentially “it would probably be easier if it were written in Rust”.
Just make it part of systemd, problem solved.
Re: Hacking on PostgreSQL is hard
#25TL; DR: testing is hard, and developers aren't very good at it. It also sounds like internal documentation may be lacking, which isn't surprising.
Well, PostgreSQL has one of the best internals documentation, it is just that internals are complex, mostly by necessity.
Re: Hacking on PostgreSQL is hard
#26[flagged]
And what does „hard“ even mean? Hard for whom? Hard for the average person? For the average developer? Hard for an expert on a particular topic? Hard for someone having practiced a lot or not?
Re: Hacking on PostgreSQL is hard
#27lol, of course the first comment is essentially “it would probably be easier if it were written in Rust”.
I laughed at that, too, but the commenter, Greg Smith (disclosure: a former colleague) has been involved with Postgres for ages, and concludes that Rust would not actually be a magic bullet here.
Re: Hacking on PostgreSQL is hard
#28Could it be that RDBMS is just inherently very complex? Reminds me of this classic comment about Oracle Database:
https://news.ycombinator.com/item?id=18442941
To quote part of it
> Oracle Database 12.2.
> It is close to 25 million lines of C code.
> What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
> Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
> Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
> The only reason why this product is still surviving and still works is due to literally millions of tests!
Re: Hacking on PostgreSQL is hard
#29It seems like having a more robust testing framework might help, akin to what SQLite uses? Obviously easier said than done, but the way he presents things it seems kind of like a push and pray environment.
PostgreSQL is an order of magnitude more complex and extensible than SQLite. Any comparison is basic at best.
I suspect it may be closer to 2 orders of magnitude than 1.
eg look at the pg docs re: at least 5 different types of indices are implemented https://www.postgresql.org/docs/current/indexes.html
Everything is like that.
(I'm not bashing sqlite, it's great software, but it has a tiny fraction of the product footprint that pg has.)
Re: Hacking on PostgreSQL is hard
#30There are some striking similarities to working on another large OSS codebase: Mozilla. (I am employed there.) We have struggled with all of these things for years, and we have a much larger pool of committers (and thus much higher variance in committer abilities). Things today are much better than they used to be, even if all of the same problems are still present to some degree.
Some of what we did might translate well to PostgreSQL, some of it won't, and much of it is probably too expensive and/or too much work. (Then again, it's work that doesn't require an inflight rocket surgeon to accomplish, which means it's doable by a much larger population of developers.)
- we've long had volunteer (and later, employee) "sheriffs" that monitor CI, know how to back things out, and over time get better at recognizing the sorts of problems that come up.
- For slow or expensive tests that don't run on every commit, they'll also take care of "backfilling" test jobs to narrow down which patch or patch stack most likely caused a problem.
- As with most CI systems, there's a staging area that gets a decent level of testing before changes are merged into the main development line.
- Feature gates for larger changes, so things can land in the mainline and be worked on there for a while, with CI regression tests running both with the feature enabled and disabled (as well as feature-specific tests when it's enabled). Good for reducing bit rot.
- Extensive fuzz testing. This would probably need to be specialized to a DB environment, since they're obviously very stateful. Various forms of snapshotting are good. For the browser (and especially the JavaScript engine I work on), it's hard to overstate just how useful this is. I would guess it could work quite well for a DB engine too.
- Lots of resources poured into test machines. With enough machines, good sheriffs, and a rich test suite, test latency doesn't matter all that much. You may not know about the problems for a day or three, but if you can depend on either getting backed out or your feature re-disabled, then you can fire and forget with no guilt. (Ok, the sheriffs will start getting snippy if you bounce a landing too many times, as is their prerogative.)
I'm guessing DB development and testing has tons of idiosyncratic difficulties, but it all sounds so familiar that I think many of the same approaches could work. The inevitable "turning the buildfarm red" should not lead to "spend[ing] the afternoon, or the evening, fixing it..." Complex software is a different beast, and it's unrealistic to expect to be able to break all features down into simple obvious changes. There's just too much going on.
(You still can't handle just anyone committing just anything at any time, though. There will always be a rate of breakage introduction that your system can handle, and it's not hard to go over it.)