A race condition in Aurora RDS • Hacker News Board

Re: A race condition in Aurora RDS

#81

Post by belter » Sat, Nov 15, 2025, 6:24 PM UTC

jansommer wrote: post #7

People who have experience with Aurora and RDS Postgres: What's your experience in terms of performance? If you dont need multi A-Z and quick failover, can you achieve better performance with RDS and e.g. gp3 64.000 iops and 3125 throughput (assuming everything else can deliver that and cpu/mem isn't the bottleneck)? Aurora seems to be especially slow for inserts and also quite expensive compared to what I get with R…

> There's an abundance of benchmarks showing Aurora is better in terms of performance but they leave out so much about their RDS config that I'm having a hard time believing them.

Do you have a problem believing these claims on equivalent hardware?: https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmo...

Or do your own performance assessments, following the published document and templates available so you can find the facts on your own?

For Aurora MySql:

"Amazon Aurora Performance Assessment Technical Guide" - https://d1.awsstatic.com/product-marketing/Aurora/RDS_Aurora...

For Aurora Postgres:

"...Steps to benchmark the performance of the PostgreSQL-compatible edition of Amazon Aurora using the pgbench and sysbench benchmarking tools..." - https://d1.awsstatic.com/product-marketing/Aurora/RDS_Aurora...

"Automate benchmark tests for Amazon Aurora PostgreSQL" - https://aws.amazon.com/blogs/database/automate-benchmark-tes...

"Benchmarking Amazon Aurora Limitless with pgbench" - https://aws.amazon.com/blogs/database/benchmarking-amazon-au...

Re: A race condition in Aurora RDS

#82

Post by evanelias » Sat, Nov 15, 2025, 8:12 PM UTC

o11c wrote: post #71

evanelias wrote: post #70

Earlier quoted context omitted.

What do you mean? Autocommit mode is the default mode in Postgres and MS SQL Server as well. This is by no means a MySQL-specific behavior! When you're in autocommit mode, BEGIN starts an explicit transaction, but after that transaction (either COMMIT or ROLLBACK), you return to autocommit mode. The situation being described upthread is a case where a transaction was started, and then rolled back by the server due to…

Yes, and most DBMS's are full of historical mistakes. In a sane world, statements outside `BEGIN` would be an unconditional error.

Lack of autocommit would be bad for performance at scale, since it would add latency to every single query. And the MVCC implications are non-trivial, especially for interactive queries (human taking their time typing) while using REPEATABLE READ isolation or stronger... every interactive query would effectively disrupt purge/vacuum until the user commits. And as the sibling comment noted, that would be quite harmful if the user completely forgets to commit, which is common.

In any case, that's a subjective opinion on database design, not a bug. Anyway it's fairly tangential to the client library bug described up-thread.

Re: A race condition in Aurora RDS

#83

Post by yencabulator » Sun, Nov 16, 2025, 8:33 PM UTC

evanelias wrote: post #52

sroussey wrote: post #49

Earlier quoted context omitted.

Converting a connection to autocommit upon error. Yikes!!

If I'm reading this correctly, it sounds like the connection was already using autocommit by default? In that situation, if you initiate a transaction, and then it gets rolled back, you're back in autocommit unless/until you initiate another transaction. If so, that part is all totally normal and expected. It's just that due to a bug in the Python client library (16 years ago), the rollback was happening silently bec…

Is there any scenario in a sane world where a transaction ceases to be in scope just because it went into an error state? I'd have expect the client to send an explicit ROLLBACK when they realize a transaction is in an error state, not for the server to end it and just notify the client. This is how psql appears to the end user.

  postgres=# begin;
  BEGIN
  postgres=*# bork;
  ERROR:  syntax error at or near "bork"
  LINE 1: bork;
          ^
  postgres=!# select 1;
  ERROR:  current transaction is aborted, commands ignored until end of transaction block
  postgres=!# rollback;
  ROLLBACK
  postgres=# select 1;
   ?column?
  ----------
          1
  (1 row)
  
  postgres=#

Re: A race condition in Aurora RDS

#84

Post by evanelias » Sun, Nov 16, 2025, 9:08 PM UTC

yencabulator wrote: post #83

evanelias wrote: post #52

Earlier quoted context omitted.

If I'm reading this correctly, it sounds like the connection was already using autocommit by default? In that situation, if you initiate a transaction, and then it gets rolled back, you're back in autocommit unless/until you initiate another transaction. If so, that part is all totally normal and expected. It's just that due to a bug in the Python client library (16 years ago), the rollback was happening silently bec…

Is there any scenario in a sane world where a transaction ceases to be in scope just because it went into an error state? I'd have expect the client to send an explicit ROLLBACK when they realize a transaction is in an error state, not for the server to end it and just notify the client. This is how psql appears to the end user. postgres=# begin; BEGIN postgres=*# bork; ERROR: syntax error at or near "bork" LINE 1: b…

Every DBMS handles errors slightly differently. In a sane world you shouldn't ever ignore errors from the database. It's unfortunate to hear that a Python MySQL client library had a bug that failed to expose errors properly in one specific situation 16 years ago, but that's not terribly relevant to today.

Postgres behavior with errors isn't even necessarily desirable -- in terms of ergonomics, why should every typo in an interactive session require me to start my transaction over from scratch?

Re: A race condition in Aurora RDS

#85

Post by yencabulator » Sun, Nov 16, 2025, 9:41 PM UTC

evanelias wrote: post #84

yencabulator wrote: post #83

Earlier quoted context omitted.

Is there any scenario in a sane world where a transaction ceases to be in scope just because it went into an error state? I'd have expect the client to send an explicit ROLLBACK when they realize a transaction is in an error state, not for the server to end it and just notify the client. This is how psql appears to the end user. postgres=# begin; BEGIN postgres=*# bork; ERROR: syntax error at or near "bork" LINE 1: b…

Every DBMS handles errors slightly differently. In a sane world you shouldn't ever ignore errors from the database. It's unfortunate to hear that a Python MySQL client library had a bug that failed to expose errors properly in one specific situation 16 years ago, but that's not terribly relevant to today. Postgres behavior with errors isn't even necessarily desirable -- in terms of ergonomics, why should every typo i…

> why should every typo in an interactive session require me to start my transaction over from scratch?

That part would hold even for the MySQL auto-rollback implied above.

Re: A race condition in Aurora RDS

#86

Post by evanelias » Sun, Nov 16, 2025, 9:55 PM UTC

yencabulator wrote: post #85

evanelias wrote: post #84

Earlier quoted context omitted.

Every DBMS handles errors slightly differently. In a sane world you shouldn't ever ignore errors from the database. It's unfortunate to hear that a Python MySQL client library had a bug that failed to expose errors properly in one specific situation 16 years ago, but that's not terribly relevant to today. Postgres behavior with errors isn't even necessarily desirable -- in terms of ergonomics, why should every typo i…

> why should every typo in an interactive session require me to start my transaction over from scratch? That part would hold even for the MySQL auto-rollback implied above.

No, the situation described upthread is about a deadlock error, not a typo. In MySQL, syntax errors throw a statement-level error but it does not affect the transaction state in any way. If you were in a transaction, you're still in a transaction after a typo.

With deadlocks in MySQL, the error you receive is "Deadlock found when trying to get lock; try restarting transaction" i.e. it specifically tells you the transaction needs to start over in that situation.

In programmatic contexts, transactions are typically represented by an object/struct type, and a correctly-implemented database driver for MySQL handles this properly and invalidates the transaction object as appropriate if the error dictates it. So this isn't really even a common foot-gun in practical terms.

Re: A race condition in Aurora RDS

#87

Post by nijave » Mon, Nov 17, 2025, 3:30 PM UTC

jansommer wrote: post #31

nijave wrote: post #22

Earlier quoted context omitted.

We've seen better results and lower costs in a 1 writer, 1-2 reader setup on Aurora PG 14. The main advantages are 1) you don't re-pay for storage for each instance--you pay for cluster storage instead of per-instance storage & 2) you no longer need to provision IOPs and it provides ~80k IOPs If you have a PG cluster with 1 writer, 2 readers, 10Ti of storage and 16k provision IOPs (io1/2 has better latency than gp3),…

Have you benchmarked your load on RDS? [0] says that IOPS on Aurora is vastly different from actual IOPS. We have just one writer instance and mostly write 100's of GB in bulk. [0] https://dev.to/aws-heroes/100k-write-iops-in-aurora-t3medium...

We didn't benchmark--we used APM data in Datadog to compare setups before and after migration

I believe the article is talking about I/O aggregate operations vs I/O average per second. I'm talking strictly about the "average per second" variety. The former is really only relevant for billing in the standard billing mode.

Actually a big motivator for the migration was batch writes (we generate tables in Snowflake, export to S3, then import from S3 using the AWS RDS extension) and Aurora (with ability to handle big spikes) helped us a lot. We'd see application performance (query latency reported by APM) increase a decent amount during these bulk imports and it was much less impactful with Aurora.

iirc it was something like 4-5ms to 10-12ms query latency for some common queries regularly and during import respectively with RDS PG and more like 6-7ms during import on Aurora (mainly because we were exhausting IO during imports before)

Re: A race condition in Aurora RDS

#88

Post by theanomaly » Mon, Nov 17, 2025, 4:43 PM UTC

belter wrote: post #77

gtowey wrote: post #3

This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…

The article is low quality. It does not mention which Aurora PostgreSQL version was involved, and it provides no real detail about how the staging environment differed from production, only saying that staging “didn’t reproduce the exact conditions,” which is not actionable. This AWS documentation section: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQ... “Amazon Aurora PostgreSQL updates”: under Aurora…

We're running Aurora PostgreSQL 15.12, which includes the fix mentioned in the release notes. Looking at this comment and the AWS documentation, I think there's an important distinction to make about what was actually fixed in Aurora PostgreSQL 15.12.4. Based on our experience and analysis, we believe AWS's fix primarily focused on data protection rather than eliminating the race condition itself.

Here's what we think is happening: Before the fix (pre-15.12.4):

1. Failover starts

2. Both instances accept and process writes simultaneously

3. Failover eventually completes after the writer steps down

4. Result: Potential data consistency issues ???

After the fix (15.12.4+):

1. Failover starts

2. If the old writer doesn't demote before the new writer is promoted, the storage layer now detects this and rejects write requests

3. Both instances restart/crash

4. Failover fails or requires manual intervention

The underlying race condition between writer demotion and reader promotion still exists - AWS just added a safety mechanism at the storage layer to prevent the dangerous scenario of two writers operating simultaneously. They essentially converted a data inconsistency risk into an availability issue. This would explain why we're still seeing failover failures on 15.12 - the race condition wasn't eliminated, just made safer.

The comment in the release notes about "fixed a race condition where an old writer instance may not step down" is somewhat misleading - it's more accurate to say they "mitigated the consequences of the race condition" by having the storage layer reject writes when it detects the problematic state and that is probably why AWS Support did not point us to this release when we raised the issue.