Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL, Memory and the Cloud

sosna.de

Post reply on HN
Print view View original

41–50 of 60 posts

Re: PostgreSQL, Memory and the Cloud

#41

Post by RNCTX »

yashap wrote: post #31
aeyes wrote: post #8

Wow, the title of this post is very calm compared to what is actually happening. CloudSQL Postgres is running with a misconfigured OS OOM killer, crashes Postmaster randomly even if memory use is below instance spec. GCP closes this bug report as "Won't fix". This is a priority 1 issue. Seeing a wontfix for this has completely destroyed my trust of their judgement. The bug report states that they have been in contact…

So many GCP products are surprisingly terrible. Certainly not all of them, some are really good, like GKE, Cloud Storage and Cloud Load Balancer. But Cloud SQL is pretty weak, and products like Cloud Logging, Cloud Metrics and Cloud Tracing are legitimately terrible. Cloud NAT is pretty sketchy too, and can lead to a lot of egress issues if not configured perfectly. My current workplace uses GCP, my last workplace us…

Not really surprising if you consider their likely motivations.

Google isn't in the business of selling things to end users, they're in the business of selling ads. The only thing GCP gives them (outside of getting wall streeters off their backs a few years ago when everyone and their brother was starting a cloud service) is a credit to their own infrastructure cost by selling excess to random joes.

Therefore I'm not surprised that AWS continues to be the defacto, they do sell things to end users. I'm not surprised that Azure is growing quickly, either, since MS also sells things to end users and they needed a way to transition their on-premise stuff to the wires.

Re: PostgreSQL, Memory and the Cloud

#42

Post by Proven »

Winsaucerer wrote: post #34

Are there any good/recommended books or resources for someone who wants to learn how to run postgresql well? E.g, what defaults to change and when, settings for the host OS (such as in the parent linked article), overall tips/insights/recommendations.

PostgreSQL docs. And google.

Re: PostgreSQL, Memory and the Cloud

#43

Post by zingar »

I'd like to thank the author for their clear, simple explanation. I haven't had to think about allocating memory since university and am not practiced thinking about it in my software but now I feel like I have useful ways to think about why processes just disappear sometimes.

Re: PostgreSQL, Memory and the Cloud

#44

Post by mixmastamyk »

Winsaucerer wrote: post #34

Are there any good/recommended books or resources for someone who wants to learn how to run postgresql well? E.g, what defaults to change and when, settings for the host OS (such as in the parent linked article), overall tips/insights/recommendations.

Postgres up and running by o'reilly is decent. Reading the docs for the config files is necessary as well.

Re: PostgreSQL, Memory and the Cloud

#45

Post by mixmastamyk »

thyrsus wrote: post #10

Are there recommendations for learning about Linux kernel memory management? Two anecdata: * I had some compute servers that were up for 200 days. The customers noticed that they were half as fast as identical hardware just booted. Dropping the file system cache ("echo 3 | sudo dd of=/proc/sys/vm/drop_cache") brought the speed back up to the newly deployed servers. WTF? File system caches are supposed to be zero cost…

Memory fragmentation? Dropping the cache and restarting high mem services at the same time might clear things up.

Re: PostgreSQL, Memory and the Cloud

#46

Post by breakingcups »

jaredsohn wrote: post #26
perlgeek wrote: post #22

Earlier quoted context omitted.

Another "fun" interaction pattern: User reports a bug (or a feature request), several others subscribe to and/or vote for this to be solved, and then a service rep closes the issue because there wasn't any recent activity. I've observed with with Atlassian where I wanted to report a Jira bug, but found that it had already been opened some years before, more than a hundred people had subscribed, bug was still closed a…

I don't think a 'me too' on this bug would be noise right now. Last activity was in 2019; having someone state the problem is still real in 2021 could impact if it gets fixed.

Not for Atlassian. I'm subscribed to several serious issues, ranging from usability and compatibility to causing actual data loss since about 2015. Each week I'll get "me too"s and "+1"s in my mailbox, but nothing from Atlassian.

Thankfully, they've deprecated their on-premise products, leading us to finally find a better alternative.

Re: PostgreSQL, Memory and the Cloud

#47

Post by ris »

Sytten wrote: post #35
aeyes wrote: post #8

Wow, the title of this post is very calm compared to what is actually happening. CloudSQL Postgres is running with a misconfigured OS OOM killer, crashes Postmaster randomly even if memory use is below instance spec. GCP closes this bug report as "Won't fix". This is a priority 1 issue. Seeing a wontfix for this has completely destroyed my trust of their judgement. The bug report states that they have been in contact…

I concur with the other comments, Cloud SQL is a very mediocre service at best. Lot's of weird issues and the engineering team doesn't seem to care. We also had the segfault due to query insight. Just the fact that you can't upgrade your database version without creating a new instance and restoring a backup is just bad. I also suggest aiven as an alternative that works very well and cost is reasonable.

> I also suggest aiven as an alternative that works very well and cost is reasonable.

Seconded. Responsive support too.

Re: PostgreSQL, Memory and the Cloud

#48

Post by ris »

jpeter wrote: post #3
yjftsjthsd-h wrote: post #2

So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?

And if there are no problems, why is it not disabled by default?

It's extremely useful to be able to map more virtual memory space than exists physical memory in your computer. This is what makes e.g. mmap'ed access to large files possible.

Re: PostgreSQL, Memory and the Cloud

#49

Post by Diggsey »

lokedhs wrote: post #40
Diggsey wrote: post #5

Earlier quoted context omitted.

Some programs allocate a lot of virtual memory and then don't use it. Also, linux's forking model can result in a lot of virtual memory being allocated if a heavy-weight program tries to fork+exec a lot of smaller programs, since fork+exec it not atomic and briefly doubles the virtual memory usage of the original program. I think there are better ways to spawn programs that don't suffer from this problem now... If yo…

The fork issue is solved by adding swap. Making sure you have plenty of swap solves these issues, and I'd like to argue that it is more reliable than using overcommit.

To be clear: I hate that overcommit exists. I think the whole situation the OOM killer makes zero sense for servers.

Having plenty of swap can help with programs that use a lot of virtual memory. AFAICT, there's no solution for the cgroups issue though.

Re: PostgreSQL, Memory and the Cloud

#50

Post by rossmohax »

yashap wrote: post #36
yashap wrote: post #31

Earlier quoted context omitted.

So many GCP products are surprisingly terrible. Certainly not all of them, some are really good, like GKE, Cloud Storage and Cloud Load Balancer. But Cloud SQL is pretty weak, and products like Cloud Logging, Cloud Metrics and Cloud Tracing are legitimately terrible. Cloud NAT is pretty sketchy too, and can lead to a lot of egress issues if not configured perfectly. My current workplace uses GCP, my last workplace us…

An example of the kinds of quality issues you run into with so many GCP products: https://github.com/googleapis/cloud-trace-nodejs/issues/1272 Basically, their Cloud Tracing product is broken for modern Node/Postgres (in terms of showing PG queries and whatnot in traces), users have found the issue (and a seemingly super simple fix), but it’s been over a year and Google still haven’t fixed it. Google’s response is “y…

> Cloud SQL isn’t terrible, but at least the Postgres version is one of the weaker managed Postgres offerings out there. And their whole observability stack (Logging/Monitoring/Tracing/Error Reporting) is legit terrible compared to competing products. Compared to other products I’ve used in the space, Cloud Logging is unbelievably worse than Sumo Logic, Cloud Metrics soooo much worse than Grafana+Prometheus, Cloud Tracing way worse than offerings from Datadog or New Relic, Cloud Error Reporting is ridiculously far behind Sentry, etc.

To be fair, AWS CloudWatch is even worse.

Post reply on HN