Yikes! This is exactly the kind of invariant I'd expect Aurora to maintain on my behalf. It is why I pay them so much...
It did, the storage layer did not allow for concurrent writes.
A race condition in Aurora RDS
61–70 of 88 posts
Re: A race condition in Aurora RDS
#62Earlier quoted context omitted.
Works fine for workloads like: 1. I need to grab some rows from a table 2. Eventual consistency is good enough And that's a lot of workloads.
That's readonly. RW workloads usually don't tolerate eventual consistency on the thing they're writing.
Yeah, if you have a mix of reads and writes in a workflow, you gotta hit the writer node. But a lot of times an endpoint is only reading data from a particular DB.
Re: A race condition in Aurora RDS
#63This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
P0 if it happens to everyone, right? Like the USE1 outage recently. If it is 0.001% of customers (enough to get a HN story) is may not be that high. Maybe this customer is on a migration or upgrade path under the hood. Or just on a bad unit in the rack.
Re: A race condition in Aurora RDS
#64I think OP is wrong in their hypothesis based on the logs they share and the root cause AWS support provided them.
I think the promotion fails to happen and then an external watchdog notices that it didn’t, and kills everything ASAP as it’s a cluster state mismatch.
The message about the storage subsystem going away is after the other Postgres process was kill -9’d.
Re: A race condition in Aurora RDS
#65People who have experience with Aurora and RDS Postgres: What's your experience in terms of performance? If you dont need multi A-Z and quick failover, can you achieve better performance with RDS and e.g. gp3 64.000 iops and 3125 throughput (assuming everything else can deliver that and cpu/mem isn't the bottleneck)? Aurora seems to be especially slow for inserts and also quite expensive compared to what I get with R…
For me, the big miss with Postgres Aurora RDS was costs. We had some queries that did a fair amount of I/O in a way that would not normally be a problem, but in the Aurora Postgres RDS world that I/O was crazy expensive. A couple of fuzzy queries blew costs up to over $3,000/month for a database that should have cost maybe $50-$100/month. And this was for a dataset of only about 15 million rows without anything crazy…
Sounds like you need to use IO optimized storage billing mode.
Re: A race condition in Aurora RDS
#66Glad to know I’m not crazy.
AWS Support initially pushed back and suggested it's because of high replication lag but they were looking at metrics that were more than 24 hours old. What kind of failure did you encounter? I really want to understand what edge case we triggered in their failover process - especially since we could not reproduce it in other regions.
My cluster recently started to failover every few days whenever it experiences the load to trigger scale up from 1-2 to 20+ acu.
And then I also encountered errors just like op in my app layer about trying to execute a write query via read-only transaction.
The workaround so far is to invalidate connection on error. When app reconnects the cluster write endpoint correctly leads to current primary.
Re: A race condition in Aurora RDS
#67Earlier quoted context omitted.
An experience not exclusive to cloud vendors :) Even better when the vendor throws their hands up cause the issue is not reliably repro'able. That was when I scripted away a test that ran hundreds of times a day on a lower environment, attempting repro. As they say, at scale, even insignificant issues become significant. I don't remember clearly, I think it was a 5-10% chance that the issue triggered. At least confir…
I've had customers with load related bugs for years simply because they'd reboot when the problem happened. When dealing with the F100 it seems there is a rather limited number of people in these organizations that can troubleshoot complex issues, that or they lock them away out of sight.
If I had a nickel for every time I had to explain that rebooting a database server is usually the wrong choice I would have quite a fortune.
Re: A race condition in Aurora RDS
#68This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
> How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? If it's anything like how Azure handles this kind of issue, it's likely "lots of people have experienced it, a restart fixes it so no one cares that much, few have any idea how to figure out a root cause on their own, and the process to find a root cause with the vendor is so painfu…
Azure yes, I'd expect this and the restart would take many minutes.
Been there done that.
AWS this is surprising
Re: A race condition in Aurora RDS
#69Earlier quoted context omitted.
Converting a connection to autocommit upon error. Yikes!!
If I'm reading this correctly, it sounds like the connection was already using autocommit by default? In that situation, if you initiate a transaction, and then it gets rolled back, you're back in autocommit unless/until you initiate another transaction. If so, that part is all totally normal and expected. It's just that due to a bug in the Python client library (16 years ago), the rollback was happening silently bec…
I would argue that it's a bug for it even to be possible to autocommit.
Re: A race condition in Aurora RDS
#70Earlier quoted context omitted.
If I'm reading this correctly, it sounds like the connection was already using autocommit by default? In that situation, if you initiate a transaction, and then it gets rolled back, you're back in autocommit unless/until you initiate another transaction. If so, that part is all totally normal and expected. It's just that due to a bug in the Python client library (16 years ago), the rollback was happening silently bec…
I would argue that it's a bug for it even to be possible to autocommit.
What do you mean? Autocommit mode is the default mode in Postgres and MS SQL Server as well. This is by no means a MySQL-specific behavior!
When you're in autocommit mode, BEGIN starts an explicit transaction, but after that transaction (either COMMIT or ROLLBACK), you return to autocommit mode.
The situation being described upthread is a case where a transaction was started, and then rolled back by the server due to deadlock error. So it's totally normal that you're back in autocommit mode after the rollback. Most DBMS handle this identically.
The bug described was entirely in the client library failing to surface the deadlock error. There's simply no autocommit-related bug as it was described.