Live data from Hacker News

Implement table partitioning

git.postgresql.org

11–20 of 60 posts

Re: Implement table partitioning

#11
Any support for "rolling" partitions? e.g. A partition for data updated less than a day ago, another for data from 2-7 days ago, etc.

I miss this from Oracle; it allows nice index optimizations as the query patterns are different for recent data vs. historical data.

I think it could be set up with a mess of triggers and a cron job... but it would be nice to have a canonical way to do this.

Re: Implement table partitioning

#12
post #4

If you also didn't know what exactly partitioned tables are, here's a nice introduction from Microsoft: https://technet.microsoft.com/en-us/library/ms190787(v=sql.1... It is for the SQL server but I assume it would be mostly relevant. Please correct me if I'm wrong.

So this is all about partitioning data into different storage files on the same server? What is the main benefit of that?

> For example, if a current month of data is primarily used for INSERT, UPDATE, DELETE, and MERGE operations while previous months are used primarily for SELECT queries, managing this table may be easier if it is partitioned by month. This benefit can be especially true if regular maintenance operations on the table only have to target a subset of the data. If the table is not partitioned, these operations can consume lots of resources on an entire data set. With partitioning, maintenance operations, such as index rebuilds and defragmentations, can be performed on a single month of write-only data, for example, while the read-only data is still available for online access.

The "General Ledger Entry" table in most accounting systems ends up being millions to billions of rows. Except for rare circumstances, prior periods are read-only due to business rules.

Re: Implement table partitioning

#13
post #4

If you also didn't know what exactly partitioned tables are, here's a nice introduction from Microsoft: https://technet.microsoft.com/en-us/library/ms190787(v=sql.1... It is for the SQL server but I assume it would be mostly relevant. Please correct me if I'm wrong.

So this is all about partitioning data into different storage files on the same server? What is the main benefit of that?

If you combine the partitions with tablespaces, you can put tables on multiple disks. Let's say you keep a record of all orders you have processed. During the day-to-day operation, you need, say, the last 2 months of data all the time, but the older data you only need for reporting here and then.

By partitioning, you can keep the recent data on a fast disk and the older data on slower disks while still being able to run reports over the whole dataset.

And once you really don't need the old data any more, you can just bulk-remove partitions which will get rid of everything in that partition without touching anything else.

Even then you don't split over tablespaces: By keeping the data that's changing often separate from the data that's static and is only read, then you gain some advantages in index management and disk load when vacuum runs as it mostly wouldn't have to touch the archive partitions.

Re: Implement table partitioning

#14
post #9

While seemingly extensive, I don't quite like the commit message. I doesn't say what TP is , and what its use cases would be. That's the first thing you should say, else how am I going to understand / keep interest in the rest of the text?

The commit is written by postgres developers for postgres developers. I would say that 90% of the intended audience of that commit message doesn't need an explanation what table partitioning does.

For them this would be needless clutter that's not at all relevant to the commit.

Once we're reaching the 10.0 release, human-friendly release notes, additional manual chapters and sample code will be written for the users to understand (in-fact, the commit linked by this submission already contains quite a bit of additional documentation to be added to the manual).

Re: Implement table partitioning

#15
About donations: I believe PostgreSQL now deserves more advertising and marketing to develop its adoption in major companies and, hence, get more funding. If I donate on the website, it says it will help conferences. Where should I donate?

Re: Implement table partitioning

#16

Any support for "rolling" partitions? e.g. A partition for data updated less than a day ago, another for data from 2-7 days ago, etc. I miss this from Oracle; it allows nice index optimizations as the query patterns are different for recent data vs. historical data. I think it could be set up with a mess of triggers and a cron job... but it would be nice to have a canonical way to do this.

Living with the cron jobs for a big mysql db, and wishing the DB understood this seemingly common use-case :(

Re: Implement table partitioning

#17
post #4

If you also didn't know what exactly partitioned tables are, here's a nice introduction from Microsoft: https://technet.microsoft.com/en-us/library/ms190787(v=sql.1... It is for the SQL server but I assume it would be mostly relevant. Please correct me if I'm wrong.

So this is all about partitioning data into different storage files on the same server? What is the main benefit of that?

Metadata operations on partitions can be very fast. One simple example is date based housekeeping. Deleting a month of data will be quite intensive on most databases, whereby dropping a partition from the table is effectively instant.

Partion switching is also fast. Say you have a summary table that is rolled up by month, but you want to recalculate the summaries every so often. You can build a month into a new table and then switch the new table for a partition in the summary table.

Re: Implement table partitioning

#18

Any support for "rolling" partitions? e.g. A partition for data updated less than a day ago, another for data from 2-7 days ago, etc. I miss this from Oracle; it allows nice index optimizations as the query patterns are different for recent data vs. historical data. I think it could be set up with a mess of triggers and a cron job... but it would be nice to have a canonical way to do this.

Living with the cron jobs for a big mysql db, and wishing the DB understood this seemingly common use-case :(

Honestly i wouldn't call it "common". It's useful, and if it existed I could see it changing how I design a database, but it's not something I can say i've ever thought about needing before.

But then again, maybe i'm the outlier here.

Re: Implement table partitioning

#19
I just tried to implement table partitioning in PostgreSQL 9.6 this week. With some triggers and check constraints this seem to work quite nicely, but I was a bit disappointed that hash based partitioning is currently not possible (at least not without extensions).

Will hash based partitioning be included in PostgreSQL 10? The post notes

  A partitioning "column" can be an expression.
so I can assume it will be supported?

Re: Implement table partitioning

#20
post #19

I just tried to implement table partitioning in PostgreSQL 9.6 this week. With some triggers and check constraints this seem to work quite nicely, but I was a bit disappointed that hash based partitioning is currently not possible (at least not without extensions). Will hash based partitioning be included in PostgreSQL 10? The post notes A partitioning "column" can be an expression. so I can assume it will be support…

Not natively, as in there is no PARTITION BY HASH (). What limitations do you face when trying to roll-your-own hash partitioning using check constraints (in 9.6)?
Post reply on HN