Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: Amazon RDS for PostgreSQL 17.4

jepsen.io

Post reply on HN
Print view View original

61–70 of 153 posts

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#61

Post by dang »

aphyr wrote: post #60
tibbar wrote: post #3

The submitted title buries the lede: RDS for PostgreSQL 17.4 does not properly implement snapshot isolation.

Folks on HN are often upset with the titles of Jepsen reports, so perhaps a little more context is in order. Jepsen reports are usually the product of a long collaboration with a client. Clients often have strong feelings about how the report is titled--is it too harsh on the system, or too favorable? Does it capture the most meaningful of the dozen-odd issues we found? Is it fair, in the sense that Jepsen aims to be…

Given that author and submitter (and commenter!) are all the same person I think we can go with your choice :)

The fact that the thread is high on HN, plus the GP comment is high in the thread, plus that the audience knows how interesting Jepsen reports get, should be enough to convey the needful.

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#62

Post by aphyr »

password4321 wrote: post #57

It would be great to get all the Amazon RDS flavors Jepsen'd.

I have actually been working on this (very slowly, in occasional nights and weekends!) Peter Alvaro and I reported on a safety issue in RDS for MySQL here too: https://jepsen.io/analyses/mysql-8.0.34#fractured-read-like-...

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#63

Post by mikesun »

aphyr wrote: post #43
ezekiel68 wrote: post #26

In my reading of this, it looks like the practical implication could be that reads happening quickly after writes to the same row(s) might return stale data. The write transaction gets marked as complete before all of the distributed layers of a multi AZ RDS instance have been fully updated, such that immediate reads from the same rows might return nothing (if the row does not exist yet) or older values if the column…

This isn't just stale data, in the sense of "a point-in-time consistent snapshot which does not reflect some recent transactions". I think what's going on here is that a read-only transaction against a secondary can observe some transaction T, but also miss transactions which must have logically executed before T.

"I think what's going on here is that a read-only transaction against a secondary can observe some transaction T, but also miss transactions which must have logically executed before T."

i was intuitively wondering the same but i'm having trouble reasoning how the post's example with transactions 1, 2, 3, 4 exhibits this behavior. in the example, is transaction 2 the only read-only transaction and therefore the only transaction to read from the read replica? i.e. transactions 1, 3, 4 use the primary and transaction 2 uses the read replica?

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#64

Post by aphyr »

nijave wrote: post #15

It's not entirely clear but this isn't an issue in multi instance upstream Postgres clusters? Am I correct in understanding either AWS is doing something with the cluster configuration or has added some patches that introduce this behavior?

This is a very good question! I do not understand AWS's replication architecture well enough to reimplement it with standard Postgres yet. This behavior doesn't happen in single-node Postgres, as far as I can tell, but it might happen in some replication setups!

I also understand there are lots of ways to do Postgres replication in general, with varying results. For instance, here's Bin Wang's report on Patroni: https://www.binwang.me/2024-12-02-PostgreSQL-High-Availabili...

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#65

Post by aphyr »

mikesun wrote: post #63
aphyr wrote: post #43

Earlier quoted context omitted.

This isn't just stale data, in the sense of "a point-in-time consistent snapshot which does not reflect some recent transactions". I think what's going on here is that a read-only transaction against a secondary can observe some transaction T, but also miss transactions which must have logically executed before T.

"I think what's going on here is that a read-only transaction against a secondary can observe some transaction T, but also miss transactions which must have logically executed before T." i was intuitively wondering the same but i'm having trouble reasoning how the post's example with transactions 1, 2, 3, 4 exhibits this behavior. in the example, is transaction 2 the only read-only transaction and therefore the only…

Yeah, that's right. It may be that the (apparent) order of transactions differs between primary and secondary.

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#66

Post by cnlwsu »

billiam wrote: post #46

New headline: AWS RDS is not CockroachDB or Spanner. And it's not trying to be.

Cockroach doesn't offer strict serializability. It has serializability with some limits depending on clock drift. Also CockroachDB does not provide linearizability over the entire database.

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#68

Post by password4321 »

aphyr wrote: post #62
password4321 wrote: post #57

It would be great to get all the Amazon RDS flavors Jepsen'd.

I have actually been working on this (very slowly, in occasional nights and weekends!) Peter Alvaro and I reported on a safety issue in RDS for MySQL here too: https://jepsen.io/analyses/mysql-8.0.34#fractured-read-like-...

There is a universe where cloud providers announce each new database offering by commissioning a Jepsen test and iterating on the results until every issue has been resolved or at least documented.

Unfortunately reliability is not that high on the priority list here. Keep up the good work!

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#69

Post by cswilliams »

Interesting. At a previous company, when we changed the pg_dump command in a backup script to start using parallel workers (-j flag) we started to rarely see errors that suggested inconsistency when restoring the backups (duplicate key errors and fk constraint errors). At the time, I tried reporting the issue to both AWS and on the Postgres mailing list but never got anywhere since I could not easily reproduce it. We eventually gave up and went back to single threaded dumps. I wonder if this issue is related to that behavior we were seeing.

Re: Jepsen: Amazon RDS for PostgreSQL 17.4

#70

Post by slt2021 »

kevincox wrote: post #52
film42 wrote: post #50

I think AWS will need to update their documentation to communicate this. Will a snapshot isolation fix introduce a performance regression in latency or throughput? Or, maybe they stand by what they have as being strong enough. Either way, they'll need to say something.

I think the ideal solution from AWS would be fixing the bug and actually providing the guarantees that the docs say that they do.

there is no trivial fix for this without breaking performance. roughly, there is no free lunch in distributed systems, and AWS made a tradeoff to relax consistency guarantees for that specific setup, and didn't really advertise that
Post reply on HN