probably should have added postgres to end of title
A race condition in Aurora RDS
41–50 of 88 posts
Re: A race condition in Aurora RDS
#42This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
> How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? If it's anything like how Azure handles this kind of issue, it's likely "lots of people have experienced it, a restart fixes it so no one cares that much, few have any idea how to figure out a root cause on their own, and the process to find a root cause with the vendor is so painfu…
Theoretically you're supposed to assign lower prio to issues with known workarounds but then there should also be reporting for product management (which assigns weight by age of first occurrence and total count of similar issues).
Amazon is mature enough for processes to reflect this, so my guess for why something like this could slip through is either too many new feature requests or many more critical issues to resolve.
Re: A race condition in Aurora RDS
#43[deleted]
Re: A race condition in Aurora RDS
#44This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
it could be most people pause writes because its going to create errors if you try and execute a write against an instance that refuses to accept and writes, and for some people those errors might not be recoverable. so they just have some option in their application that puts the application into maintenance mode where it will hard reject writes at the application layer.
Re: A race condition in Aurora RDS
#45People who have experience with Aurora and RDS Postgres: What's your experience in terms of performance? If you dont need multi A-Z and quick failover, can you achieve better performance with RDS and e.g. gp3 64.000 iops and 3125 throughput (assuming everything else can deliver that and cpu/mem isn't the bottleneck)? Aurora seems to be especially slow for inserts and also quite expensive compared to what I get with R…
For me, the big miss with Postgres Aurora RDS was costs. We had some queries that did a fair amount of I/O in a way that would not normally be a problem, but in the Aurora Postgres RDS world that I/O was crazy expensive. A couple of fuzzy queries blew costs up to over $3,000/month for a database that should have cost maybe $50-$100/month. And this was for a dataset of only about 15 million rows without anything crazy in them.
Re: A race condition in Aurora RDS
#46Am I the only one who misread that as “AI race condition”?
Re: A race condition in Aurora RDS
#47This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
Agreed, we've been running multiple aurora clusters in production for years now and have not encountered this issue with failovers.
Same. There’s something missing here.
Re: A race condition in Aurora RDS
#48Earlier quoted context omitted.
Works fine for workloads like: 1. I need to grab some rows from a table 2. Eventual consistency is good enough And that's a lot of workloads.
As a user, I've come to realize the situations where I think eventual consistency (or delayed processing) are good enough aren't the same as the folks developing most products. Nothing annoys me more than stuff not showing up immediately or having to manually refresh.
I think the key here is just proper notifications. Yes it's eventually consistent, but having a "processing" or "update in progress" is a huge improvement over showing a user old data.
Re: A race condition in Aurora RDS
#49This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
> - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? I know that there is no comparison in the user base, but a few years ago I ran into a massive Python + MySQL bug that: 1. made SELECT ... FOR UPDATE fail silenty 2. aborted the transaction and set the connection into autocommit mode This basically a worst case scenario in a transa…
Converting a connection to autocommit upon error. Yikes!!
Re: A race condition in Aurora RDS
#50Earlier quoted context omitted.
> How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? If it's anything like how Azure handles this kind of issue, it's likely "lots of people have experienced it, a restart fixes it so no one cares that much, few have any idea how to figure out a root cause on their own, and the process to find a root cause with the vendor is so painfu…
An experience not exclusive to cloud vendors :) Even better when the vendor throws their hands up cause the issue is not reliably repro'able. That was when I scripted away a test that ran hundreds of times a day on a lower environment, attempting repro. As they say, at scale, even insignificant issues become significant. I don't remember clearly, I think it was a 5-10% chance that the issue triggered. At least confir…
I've had customers with load related bugs for years simply because they'd reboot when the problem happened. When dealing with the F100 it seems there is a rather limited number of people in these organizations that can troubleshoot complex issues, that or they lock them away out of sight.