Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Subtly Bad Things Linux May Be Doing To PostgreSQL

rhaas.blogspot.in

Post reply on HN
Print view View original

11–20 of 95 posts

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#11

Post by kev009 »

contingencies wrote: post #10

Not to detract from the very intelligent and reasoned posting, but what tiny percentage of people honestly still use fat-ass RDBMS as their primary datastore and would be better off performance tuning it at the kernel IO level than actually analyzing their load and subsequently sharding or migrating their data structures to less behemoth-like datastores? Yes , RDBMS are easy to hire developers and DBAs for, are well…

That's a very obtuse point of view. I'm curious sociologically: what field do you work in and what your exposure to data is?

Consider an inventory system for a big box retailer. I can't think of anything better than a fat-ass RDBMS as the primary data store. Sharding sounds like a horrific idea. There are myriad workloads like this.

Personally, I've seen pgsql handle terabytes of data just fine and it wasn't really noteworthy or a source of problems to even bring up considering something else. YMMV but it's a good idea to use logic and reason to dictate architecture instead of following the shiny thing or hubris.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#12

Post by DrJokepu »

treenyc wrote: post #7
forkandwait wrote: post #2

Anybody know of a FreeBSD comparison?

FreeBSD has something like SHMMAXPGS, SHMMAX in postgresql.conf in the sysctl.conf kern.ipc.shmall kern.ipc.shmall

Not relevant since PostgreSQL 9.3 as PostgreSQL now uses Posix shared memory / mmap rather that System V.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#13

Post by davidw »

kev009 wrote: post #11
contingencies wrote: post #10

Not to detract from the very intelligent and reasoned posting, but what tiny percentage of people honestly still use fat-ass RDBMS as their primary datastore and would be better off performance tuning it at the kernel IO level than actually analyzing their load and subsequently sharding or migrating their data structures to less behemoth-like datastores? Yes , RDBMS are easy to hire developers and DBAs for, are well…

That's a very obtuse point of view. I'm curious sociologically: what field do you work in and what your exposure to data is? Consider an inventory system for a big box retailer. I can't think of anything better than a fat-ass RDBMS as the primary data store. Sharding sounds like a horrific idea. There are myriad workloads like this. Personally, I've seen pgsql handle terabytes of data just fine and it wasn't really n…

Well, yes and no.

Everybody knows that relational databases don't scale because they use JOINs and write to disk.

Also, relational databases weren't built for web scale. MongoDB handles web scale. You turn it on and it scales right up.

And before you knock shards, shards are the secret ingredient in the web scale sauce. They just work.

Furthermore, relational databases have impetus mismatch, and Postgresql is slow as a dog. MongoDB will run circles around Postgresql because MongoDB is web scale.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#14

Post by ars »

davidpardo wrote: post #5

Being a user of both Linux and PostrgreSQL, I'm very interested in this issue, but I only understand some of the words... Could everybody wiser than me tell me if I should be concerned and the possible implications of these decisions? Should I invest in alternative platforms?

These are very minor issues and will be noticed by very few.

If you are running a huge database and need every possible bit of performance this will matter, otherwise it's not something to worry about.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#16

Post by [deleted] »

kev009 wrote: post #11
contingencies wrote: post #10

Not to detract from the very intelligent and reasoned posting, but what tiny percentage of people honestly still use fat-ass RDBMS as their primary datastore and would be better off performance tuning it at the kernel IO level than actually analyzing their load and subsequently sharding or migrating their data structures to less behemoth-like datastores? Yes , RDBMS are easy to hire developers and DBAs for, are well…

That's a very obtuse point of view. I'm curious sociologically: what field do you work in and what your exposure to data is? Consider an inventory system for a big box retailer. I can't think of anything better than a fat-ass RDBMS as the primary data store. Sharding sounds like a horrific idea. There are myriad workloads like this. Personally, I've seen pgsql handle terabytes of data just fine and it wasn't really n…

[deleted]

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#17

Post by contingencies »

kev009 wrote: post #11
contingencies wrote: post #10

Not to detract from the very intelligent and reasoned posting, but what tiny percentage of people honestly still use fat-ass RDBMS as their primary datastore and would be better off performance tuning it at the kernel IO level than actually analyzing their load and subsequently sharding or migrating their data structures to less behemoth-like datastores? Yes , RDBMS are easy to hire developers and DBAs for, are well…

That's a very obtuse point of view. I'm curious sociologically: what field do you work in and what your exposure to data is? Consider an inventory system for a big box retailer. I can't think of anything better than a fat-ass RDBMS as the primary data store. Sharding sounds like a horrific idea. There are myriad workloads like this. Personally, I've seen pgsql handle terabytes of data just fine and it wasn't really n…

Sure, if you are running stats across everything in a nontrivial and frequently changing way, then you have a great ally in an RDBMS. But I don't believe many people do that, because usually that sort of stuff is pretty damn predictable, executed offline, or can be consolidated from shards.

However, if you have any of the following: (1) vastly different security requirements for different parts of your datastore (2) vastly different backup schedules or temporal sensitivities (3) privacy requirements deriving from different legal jurisdictions (4) wish to scale by running on commodity hardware (5) cannot tolerate any downtime whatsoever ... and probably many other cases ... then in my experience you are going to meet some serious issues with conventional RDBMS, at least with the vast majority of configurations.

I'm all for logic and reason too... but your comments seem closer to name-calling and a single example.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#18

Post by Confusion »

contingencies wrote: post #10

Not to detract from the very intelligent and reasoned posting, but what tiny percentage of people honestly still use fat-ass RDBMS as their primary datastore and would be better off performance tuning it at the kernel IO level than actually analyzing their load and subsequently sharding or migrating their data structures to less behemoth-like datastores? Yes , RDBMS are easy to hire developers and DBAs for, are well…

Don't feed the troll.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#19

Post by rimantas »

davidw wrote: post #13
kev009 wrote: post #11

Earlier quoted context omitted.

That's a very obtuse point of view. I'm curious sociologically: what field do you work in and what your exposure to data is? Consider an inventory system for a big box retailer. I can't think of anything better than a fat-ass RDBMS as the primary data store. Sharding sounds like a horrific idea. There are myriad workloads like this. Personally, I've seen pgsql handle terabytes of data just fine and it wasn't really n…

Well, yes and no. Everybody knows that relational databases don't scale because they use JOINs and write to disk. Also, relational databases weren't built for web scale. MongoDB handles web scale. You turn it on and it scales right up. And before you knock shards, shards are the secret ingredient in the web scale sauce. They just work. Furthermore, relational databases have impetus mismatch, and Postgresql is slow as…

Check this "web scale" out: http://smalldatum.blogspot.com (via http://dom.as/2014/03/31/mongo-io/ )

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#20

Post by pjc50 »

treenyc wrote: post #8
davidpardo wrote: post #5

Being a user of both Linux and PostrgreSQL, I'm very interested in this issue, but I only understand some of the words... Could everybody wiser than me tell me if I should be concerned and the possible implications of these decisions? Should I invest in alternative platforms?

same here. Was a FreeBSD user, but find hard to find VM a few years ago that support FreeBSD, may switch back, if this issue is NOT addressed.

If you're running in a VM, the VM may impose its own constraints on the disk write and virtual memory issues.
Post reply on HN