51–60 of 88 posts
probably should have added postgres to end of title
A lot of people seem to have the misconception that "Aurora" is its own unique database system, with different front-ends "pretending" to be Postgres or MySQL, but that isn't the case at all.
Earlier quoted context omitted.
> - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? I know that there is no comparison in the user base, but a few years ago I ran into a massive Python + MySQL bug that: 1. made SELECT ... FOR UPDATE fail silenty 2. aborted the transaction and set the connection into autocommit mode This basically a worst case scenario in a transa…
Converting a connection to autocommit upon error. Yikes!!
If so, that part is all totally normal and expected. It's just that due to a bug in the Python client library (16 years ago), the rollback was happening silently because the error was not surfaced properly by the client library.
> AWS has indicated a fix is on their roadmap, but as of now, the recommended mitigation aligns with our solution: use Aurora’s Failover feature on an as-needed basis and ensure that no writes are executed against the DB during the failover. Is there a case number where we can reach out to AWS regarding this recommendation?
We use Aurora MySQL but I would like to be able to point to that and ask if it applies to us.
People who have experience with Aurora and RDS Postgres: What's your experience in terms of performance? If you dont need multi A-Z and quick failover, can you achieve better performance with RDS and e.g. gp3 64.000 iops and 3125 throughput (assuming everything else can deliver that and cpu/mem isn't the bottleneck)? Aurora seems to be especially slow for inserts and also quite expensive compared to what I get with R…
We have some clusters with very high write IOPS on Aurora.
When looking at costs we modelled running MySQL and regular RDS MySQL.
We found for the IOPS capacity of Aurora we wouldn't be able to match it on AWS without paying a stupid amount more.
Earlier quoted context omitted.
This is super confusing. Check out the RDS Postgres calculator with gp3: > General Purpose SSD (gp3) - Throughput > gp3 supports a max of 4000 MiBps per volume But the docs say 2000. Then there's IOPS... The calculator allows up to 64.000 but on [0], if you expand "Higher performance and throughout" it says > Customers looking for higher performance can scale up to 80,000 IOPS and 2,000 MiBps for an additional fee. […
RDS PG stripes multiple gp3 volumes so that's why RDS throughput is higher than gp3 I think 80k IOPs on gp3 is a newer release so presumably AWS hasn't updated RDS from the old max of 64k. iirc it took a while before gp3 and io2 were even available for RDS after they were released as EBS options Edit: Presumably it takes some time to do testing/optimizations to make sure their RDS config can achieve the same performa…
This article seems to indicate that manually triggered failovers will always fail if your application tries to maintain its normal write traffic during that process. Not that I'm discounting the author's experience, but something doesn't quite add up: - How is it possible that other users of Aurora aren't experiencing this issue basically all the time? How could AWS not know it exists? - If they know, how is this not…
Wow. This is alarming. We have done a similar operation routinely on databases under pretty write intensive workloads (like 10s of thousands of inserts per second). It is so routine we have automation to adjust to planned changes in volume and do so a dozen times a month or so. It has been very robust for us. Our apps are designed for it and use AWS’s JDBC wrapper. Just one more thing to worry about I guess…
Earlier quoted context omitted.
An experience not exclusive to cloud vendors :) Even better when the vendor throws their hands up cause the issue is not reliably repro'able. That was when I scripted away a test that ran hundreds of times a day on a lower environment, attempting repro. As they say, at scale, even insignificant issues become significant. I don't remember clearly, I think it was a 5-10% chance that the issue triggered. At least confir…
I've had customers with load related bugs for years simply because they'd reboot when the problem happened. When dealing with the F100 it seems there is a rather limited number of people in these organizations that can troubleshoot complex issues, that or they lock them away out of sight.
With all the enterprise solutions being distributed, loosely coupled, self-healing, redundant, and fault-tolerant, issues like this essentially just slot in perfectly. Compound this with man-hours (especially expert ones) being a lot harder to justify for any one particular bump in tail latency, and the equation is just really not there for all this.
What gets us specifically to look into things is either the issue being operationally gnarly (e.g. frequent, impacting, or both), or management being swayed enough by principled thinking (or at least pretending to be). I'd imagine it's the same elsewhere. The latter would mostly happen if fixing a given thing becomes an office political concern, or a corporate reputation one. You might wonder if those individual issues ever snowballed into a big one, but turns out human nature takes care of that just "sufficiently enough" before it would manifest "too severely". [0]
Otherwise, you're looking at fixing / RCA'ing / working around someone else's product defect on their behalf, and giving your engineers a "fun challenge". Fun doesn't pay the bills, and we rarely saw much in return from the vendor in exchange for our research. I'd love to entertain the idea that maybe behind closed doors the negotiations went a little better because of these, but for various reasons, I really doubt so in hindsight.
[0] as delightfully subjective as those get of course
They made the storage layer in total isolation, and they made sure that it guaranteed correctness for exclusive writer access. When the upstream service failed to also make its own guarantees, the data layer was still protected.
Good job AWS engineering!