Live data from Hacker News

How Postgres stores data on disk – this one's a page turner

drew.silcock.dev

11–20 of 98 posts

Re: How Postgres stores data on disk – this one's a page turner

#11

Earlier quoted context omitted.

The submitter can edit the title after it has been auto-modified. It doesn't take a mod to do it unless some amount of time has passed.

Can't see any way of modifying the title, unless I'm missing something – maybe I missed the edit window.

Yeah, after an hour, you've probably missed it. (Source: I frequently edit titles of posts I submit to HN because HN changed it to nonsense. :) )

Re: How Postgres stores data on disk – this one's a page turner

#13
> This process of retrieving the expected database state from the WAL is called logical decoding and Postgres stores files related to this process in here.

While logical decoding is about WAL, it is not related to the recovery process. Logical decoding is a mechanism to convert the WAL entries back into the high-level operations that caused the WAL entries, for example for replication or audit.

Re: How Postgres stores data on disk – this one's a page turner

#15

A bit of curiosity: how did Postgres choose 8k pages? shouldn’t it be the FS page size to help with atomicity?

Depends very much on how the SSDs are designed internally. I think these days we have to settle for "can never be sure" of the real page size for atomicity. Pages can also become corrupt in other ways.

It is weird that "--data-checksums" isn't the default for new databases, even when it cost a bit in performance. Integrity should be more important than performance.

Re: How Postgres stores data on disk – this one's a page turner

#16

Earlier quoted context omitted.

Can't see any way of modifying the title, unless I'm missing something – maybe I missed the edit window.

Yeah, after an hour, you've probably missed it. (Source: I frequently edit titles of posts I submit to HN because HN changed it to nonsense. :) )

Thanks for letting me know, I'll make sure to edit it quickly next time :-)

Re: How Postgres stores data on disk – this one's a page turner

#17

A bit of curiosity: how did Postgres choose 8k pages? shouldn’t it be the FS page size to help with atomicity?

Depends very much on how the SSDs are designed internally. I think these days we have to settle for "can never be sure" of the real page size for atomicity. Pages can also become corrupt in other ways. It is weird that "--data-checksums" isn't the default for new databases, even when it cost a bit in performance. Integrity should be more important than performance.

Was thinking the same thing when I saw those zeros in the checksum field. Perhaps the consequences are significant.

Here's a benchmarking exercise I found: https://www-staging.commandprompt.com/uploads/images/Command...

With a tidy summary:

> Any application with a high shared buffers hit ratio: little difference. > Any application with a high ratio of reads/writes: little difference. > Data logging application with a low ratio of reads/inserts, and few updates and deletes: little difference. > Application with an equal ratio of reads/inserts, or many updates or deletes, and a low shared buffers hit ratio (for example, an ETL workload), especially where the rows are scattered among disk pages: expect double or greater CPU and disk I/O use. > Run pg_dump on a database where all rows have already been previously selected by applications: little difference. > Run pg_dump on a database with large quantities of rows inserted to insert-only tables: expect roughly double CPU and disk I/O use.

Re: How Postgres stores data on disk – this one's a page turner

#18

A bit of curiosity: how did Postgres choose 8k pages? shouldn’t it be the FS page size to help with atomicity?

Depends very much on how the SSDs are designed internally. I think these days we have to settle for "can never be sure" of the real page size for atomicity. Pages can also become corrupt in other ways. It is weird that "--data-checksums" isn't the default for new databases, even when it cost a bit in performance. Integrity should be more important than performance.

If physical integrity is already provided by the backing filesystem, such as ZFS, wouldn't --data-checksums be redundant?

Re: How Postgres stores data on disk – this one's a page turner

#19

A bit of curiosity: how did Postgres choose 8k pages? shouldn’t it be the FS page size to help with atomicity?

8k is a very common page size, but 4k isn't unheard of. Oracle's default is 4k.

The issue is that page size caps row size (for on-row storage). Also, if you have a smart clustering index, larger pages can be more efficient use of index addressing. So it's a trade-off.

Re: How Postgres stores data on disk – this one's a page turner

#20
post #13

> This process of retrieving the expected database state from the WAL is called logical decoding and Postgres stores files related to this process in here. While logical decoding is about WAL, it is not related to the recovery process. Logical decoding is a mechanism to convert the WAL entries back into the high-level operations that caused the WAL entries, for example for replication or audit.

Very good point, I've rephrased this.
Post reply on HN