Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The four horsemen behind Postgres outages

malisper.me

Post reply on HN
Print view View original

11–19 of 19 posts

Re: The four horsemen behind Postgres outages

#11

Post by dmitrijbelikov »

studmuffin650 wrote: post #7
dmitrijbelikov wrote: post #2

If you understand Postgres's problems so well (though in reality, such problems can occur in many applications, and it's unclear how your rewritten version attempts to solve them), then why not address them in the main branch by becoming a contributor. Less code and more value.

I think the issue isn’t a lack of want for fixing these issues, but the inertia and friction of getting them fixed. The issues around the transaction ids and process per connection are well known, but the changes to the codebase to fix them would either constitute a backwards incompatible change that would change storage needs or an incredibly large rewrite of the codebase that breaks with decades of assumptions. The…

TID (XID) - VACUUM, AVTOVACUUM (configuration), process per connection - pgbouncer, odyssey

Re: The four horsemen behind Postgres outages

#13

Post by titanomachy »

bananamogul wrote: post #8

This article states multiple times that spinning up a process is expensive/very expensive. Is that really true? I ask out of ignorance. “Compared to other ways of doing parallelism, processes are very expensive, both in terms of taking CPU resources but also the amount of time it takes to spin up a new process.” “Because it’s expensive to spin up new processes, Postgres will only do this for long-running queries.” Al…

It’s also important to distinguish between an os thread and a userspace thread. The author said “thread” without qualification, so I don’t know which he meant. Userspace threads can be many times lighter than processes.

Re: The four horsemen behind Postgres outages

#14

Post by guenthert »

dudul wrote: post #9
bananamogul wrote: post #8

This article states multiple times that spinning up a process is expensive/very expensive. Is that really true? I ask out of ignorance. “Compared to other ways of doing parallelism, processes are very expensive, both in terms of taking CPU resources but also the amount of time it takes to spin up a new process.” “Because it’s expensive to spin up new processes, Postgres will only do this for long-running queries.” Al…

A process is significantly more expensive than a thread on a Linux system. The main cost is memory. Processes own their memory which forces the system to duplicate resources, as opposed to threads that are able to share resources. Where have you red that "the cost between a process and a thread is relatively small."? I would be curious to see a link because the most cursory internet search would show you that it is n…

"significantly more expensive" and "relatively small"

Well, both are true of course. You will have to put numbers on those to make sensible decisions.

Historically, creating processes in Linux has been cheaper than in other Unix-like OS (you might find results of an old Byte benchmark) and much cheaper than in Windows. Creating a thread will be undoubtedly cheaper still, but whether that will change the user experience or is worth the cost depends on details (how large the memory footprint is, pagesize, how many files are open, how many threads are to be executed, their lifetime, how many execution units are available, etc.).

Re: The four horsemen behind Postgres outages

#16

Post by ubercore »

titanomachy wrote: post #13
bananamogul wrote: post #8

This article states multiple times that spinning up a process is expensive/very expensive. Is that really true? I ask out of ignorance. “Compared to other ways of doing parallelism, processes are very expensive, both in terms of taking CPU resources but also the amount of time it takes to spin up a new process.” “Because it’s expensive to spin up new processes, Postgres will only do this for long-running queries.” Al…

It’s also important to distinguish between an os thread and a userspace thread. The author said “thread” without qualification, so I don’t know which he meant. Userspace threads can be many times lighter than processes.

I think tokio does userspace threads running on a thread per core. At least by default.

Re: The four horsemen behind Postgres outages

#17

Post by yolobolo »

hoppp wrote: post #4
zaik wrote: post #3

Earlier quoted context omitted.

It is explained in the article: Any of those changes require massive or incompatible changes.

So they want 100% compatibility first and then implement features that require massive and incompatible changes?

Sure, why not? Never reaching 100% compatible is a problem as is never amounting to more than a bug for bug clone. If they can quantify exactly what they have changed for good reason then that's good for them.

Re: The four horsemen behind Postgres outages

#18

Post by hoppp »

yolobolo wrote: post #17
hoppp wrote: post #4

Earlier quoted context omitted.

So they want 100% compatibility first and then implement features that require massive and incompatible changes?

Sure, why not? Never reaching 100% compatible is a problem as is never amounting to more than a bug for bug clone. If they can quantify exactly what they have changed for good reason then that's good for them.

I applaud the idea to try to do this but for one postgresql tests are not 100% coverage.

So even if they reach 100% compatibility they never get equality.

Then Vibe into that and add changes that breaks the existing coverage...

Why not reimplement the entire thing from the ground up with the changes in mind from the start instead?

The upgrade is an afterthought to a system that is not verified to be fully working and the new features will break the existing verification too.

If I had OCD it would be triggered to the max by this. But I'm just sipping a drink from the sidelines.

Re: The four horsemen behind Postgres outages

#19

Post by yolobolo »

hoppp wrote: post #18
yolobolo wrote: post #17

Earlier quoted context omitted.

Sure, why not? Never reaching 100% compatible is a problem as is never amounting to more than a bug for bug clone. If they can quantify exactly what they have changed for good reason then that's good for them.

I applaud the idea to try to do this but for one postgresql tests are not 100% coverage. So even if they reach 100% compatibility they never get equality. Then Vibe into that and add changes that breaks the existing coverage... Why not reimplement the entire thing from the ground up with the changes in mind from the start instead? The upgrade is an afterthought to a system that is not verified to be fully working and…

Sure, I'm impressed that something like this that would have been a thought experiment or a decades long project with minimal coverage is now a prototype that seems to me like it will contribute something back even if there might not be a rush to use it directly. During the NoSQL phase one can say it was real human engineers making that stuff but often the overall quality was bellow anything that should pass a quantitative measure like a postgres test suite and people put those in production even when they were known to have deep design flaws.
Post reply on HN