Coming at this from a naive outsider perspective, the central problem described in the post (commits to PostgreSQL frequently have serious defects which must be addressed in follow-up commits) seems like one that would ideally be addressed with automated testing and CI tooling. What kind of testing does the Postgres project have? Are there tests which must pass before a commit can be integrated in the main branch? Are there tests that are only run nightly? Is most core functionality covered by quick-running unit tests, or are there significant pieces which can only be tested by hours-long integration tests? How expensive is it, in machine-hours, to run the full test suite, and how often is this done? What kinds of requirements are in place for including new tests with new code?
Hacking on PostgreSQL is hard
11–20 of 56 posts
Re: Hacking on PostgreSQL is hard
#12Coming at this from a naive outsider perspective, the central problem described in the post (commits to PostgreSQL frequently have serious defects which must be addressed in follow-up commits) seems like one that would ideally be addressed with automated testing and CI tooling. What kind of testing does the Postgres project have? Are there tests which must pass before a commit can be integrated in the main branch? Ar…
Most database companies run only a small amount of tests before committing. After committing, you run tests for thousands of hours. It sucks. You probably do this all day every day. You just run the tests on whatever you have currently committed. you kind of have to be careful about not adding more tests that make it take much much longer.
See https://news.ycombinator.com/item?id=18442941
Re: Hacking on PostgreSQL is hard
#13Coming at this from a naive outsider perspective, the central problem described in the post (commits to PostgreSQL frequently have serious defects which must be addressed in follow-up commits) seems like one that would ideally be addressed with automated testing and CI tooling. What kind of testing does the Postgres project have? Are there tests which must pass before a commit can be integrated in the main branch? Ar…
I would also note that the fix prs started landing the day after the initial commit, and other issues noted had fixes within three weeks. And of course postgresql has testing, but at universal distribution and use cases on things that will test both scheduler, network, fs, io drivers (Linux kernel, postgresql, etc, among others), some things need wider audiences or more extreme testing scenarios (SQLite for a strict subset of those considerations), and project health is measured by responding to that in a timely fashion. Afaics this is all about trunk/main, versus releases as well. So while its labeled its hard on the post (from a long time pg contributor), and yeah i might agree (cause maintainer on other software, so yeah all this resonates heavily), I’d also say its an example of things done right.
Seems like a reason to celebrate the open source model, and specifically here on how to do things better. Not to detract from universal issues for any project on maintainer availability. But, imagine a non oss database vendor with that degree of transparency or velocity, i can’t think of any that are doing anything close unless they got popped on a remote cve, aka prioritized above features or politics on a corporate dev sprint. Aka all software has bugs, it’s about how fast things are fixed, and in the context of oss imho fostering evolution among a diverse set of maintainers and use cases seems to be a better way.
As another example of that, ‘twas a PostgreSQL hacker at MS, that prevented Libxz from going wide because of caring due to perf regression and doing the analysis.
Re: Hacking on PostgreSQL is hard
#14It seems like having a more robust testing framework might help, akin to what SQLite uses?
Obviously easier said than done, but the way he presents things it seems kind of like a push and pray environment.
Re: Hacking on PostgreSQL is hard
#15TL; DR: testing is hard, and developers aren't very good at it.
It also sounds like internal documentation may be lacking, which isn't surprising.
Re: Hacking on PostgreSQL is hard
#16What's missing in this post is a deep analysis of what the bugs are and what was causing them, in a 5 times why sense. Especially if they all seem dumb stuff at first.
There are some deep lessons about programming in this Factorio Friday Facts:
https://factorio.com/blog/post/fff-366
and I wonder if postgres doesn't look like fig.1 from this blog post, before the refactoring.
Re: Hacking on PostgreSQL is hard
#17It seems like having a more robust testing framework might help, akin to what SQLite uses? Obviously easier said than done, but the way he presents things it seems kind of like a push and pray environment.
I don't think that's fair. The PostgreSQL codebase includes a lot of stuff that isn't included in SQLite, where it's covered by 3rd party projects - I'd bet their quality is nothing like SQLite or PostgreSQL. Tom Lane was involved in these commits and sounds like some of this got by him. His comments have frequently been described as "complete technical manuals", so I think that speaks to the complexity.
Re: Hacking on PostgreSQL is hard
#18It seems like having a more robust testing framework might help, akin to what SQLite uses? Obviously easier said than done, but the way he presents things it seems kind of like a push and pray environment.
PostgreSQL has different execution paths depending on load, table size, hardware etc which also complicates testing and test setup.
Re: Hacking on PostgreSQL is hard
#19Re: Hacking on PostgreSQL is hard
#20It seems like having a more robust testing framework might help, akin to what SQLite uses? Obviously easier said than done, but the way he presents things it seems kind of like a push and pray environment.
PostgreSQL is an order of magnitude more complex and extensible than SQLite.
Any comparison is basic at best.