Just make it scale: An Aurora DSQL story
allthingsdistributed.com
Just make it scale: An Aurora DSQL story
1–10 of 42 posts
Re: Just make it scale: An Aurora DSQL story
#2Early dsql had some weird limits I think - anyone actually using in production with feedback on current corners and limits?
Re: Just make it scale: An Aurora DSQL story
#3Early dsql had some weird limits I think - anyone actually using in production with feedback on current corners and limits?
I don't use it, but have been keeping an eye on it.
At launch, they limited the number of affected tuples to 10000, including tuples in secondary indexes. They recently changed this limit to:
> A transaction cannot modify more than 3,000 rows. The number of secondary indexes does not influence this number. This limit applies to all DML statements (INSERT, UPDATE, DELETE).
There are a lot of other (IMO prohibitive) restrictions listed in their docs.
https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
Re: Just make it scale: An Aurora DSQL story
#4Early dsql had some weird limits I think - anyone actually using in production with feedback on current corners and limits?
Who would use Preview products in production? I'm building out some software that would fit perfectly into the constraints set for DSQL, but I realistically can't commit to something with no pricing / guarantees.
Re: Just make it scale: An Aurora DSQL story
#5Good read. I like the part that both writing low level as well as high level component in Rust was proven worthwhile.
Maybe one can transform slow code from high level languages to low level language via LLMs in future. That can be nice performance boost for those who don't have Amazon engineers and budgets
Re: Just make it scale: An Aurora DSQL story
#6Early dsql had some weird limits I think - anyone actually using in production with feedback on current corners and limits?
Which features would you like to see the team build first? Which limits would you like to see lifted first?
Most of the limitations you can see in the documentation are things we haven't gotten to building yet, and it's super helpful to know what folks need so we can prioritize the backlog.
Re: Just make it scale: An Aurora DSQL story
#7Good read. I like the part that both writing low level as well as high level component in Rust was proven worthwhile. Maybe one can transform slow code from high level languages to low level language via LLMs in future. That can be nice performance boost for those who don't have Amazon engineers and budgets
> Maybe one can transform slow code from high level languages to low level language
I think you are describing a compiler?
Re: Just make it scale: An Aurora DSQL story
#8Many interesting things, for instance, I've been hearing a lot about how fast Java is, that it can be as fast as C++, and then I see this:
> But after a few weeks, it compiled and the results surprised us. The code was 10x faster than our carefully tuned Kotlin implementation – despite no attempt to make it faster. To put this in perspective, we had spent years incrementally improving the Kotlin version from 2,000 to 3,000 transactions per second (TPS). The Rust version, written by Java developers who were new to the language, clocked 30,000 TPS.
I feel like there is more to this, like some kind of a bottleneck, memory footprint, some IO overhead?
> Our conclusion was to rewrite our data plane entirely in Rust.
The point is well taken, figuring it out is not worth it, if you can just "rewrote" or have green field projects.
> These extension points are part of Postgres’ public API, allowing you to modify behavior without changing core code
Also, interesting. So PostgreSQL evolved to the point that it has a stable API for extensibility? This great for the project, maintain a modular design, and some stable APIs and, you can let people mix and match and reduce duplication of effort.
Re: Just make it scale: An Aurora DSQL story
#9Good read. I like the part that both writing low level as well as high level component in Rust was proven worthwhile. Maybe one can transform slow code from high level languages to low level language via LLMs in future. That can be nice performance boost for those who don't have Amazon engineers and budgets
> Maybe one can transform slow code from high level languages to low level language via LLMs in future.
This is one of the areas I'm most excited for LLM developer tooling. Choosing a language, database, or framework is a really expensive up-front decision for a lot of teams, made when they have the least information about what they're building, and very expensive to take back.
If LLM-powered tools could take 10-100x off the cost of these migrations, it would significantly reduce the risk of early decisions, and make it a ton easier to make software more reliable and cheaper to run.
It's very believable to me that, even with today's model capabilities, that 10-100x is achievable.
Re: Just make it scale: An Aurora DSQL story
#10The JVM can relocate memory to avoid fragmentation. Rust can't, at least natively. Are they not worried about this regression?