Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just make it scale: An Aurora DSQL story

allthingsdistributed.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Just make it scale: An Aurora DSQL story

#11

Post by geodel »

SahAssar wrote: post #7
geodel wrote: post #5

Good read. I like the part that both writing low level as well as high level component in Rust was proven worthwhile. Maybe one can transform slow code from high level languages to low level language via LLMs in future. That can be nice performance boost for those who don't have Amazon engineers and budgets

> Maybe one can transform slow code from high level languages to low level language I think you are describing a compiler?

I mean reading this article:

1) Kotlin code --> Java byte code --> JVM execution (slow)

vs

2) Kotin code --> Rust/Zig code --> Zig compiler --> native execution (fast)

Compiler is involved in both cases but I was thinking of 2) where slower code in high level lang is converted to another lang code. The compiler of which is known to produce fast runinng code.

Re: Just make it scale: An Aurora DSQL story

#12

Post by anarazel »

Demiurge wrote: post #8

Many interesting things, for instance, I've been hearing a lot about how fast Java is, that it can be as fast as C++, and then I see this: > But after a few weeks, it compiled and the results surprised us. The code was 10x faster than our carefully tuned Kotlin implementation – despite no attempt to make it faster. To put this in perspective, we had spent years incrementally improving the Kotlin version from 2,000 to…

> So PostgreSQL evolved to the point that it has a stable API for extensibility?

Not across major versions, no. I seriously doubt we will ever make promises around that. It would hamper development way too much.

Re: Just make it scale: An Aurora DSQL story

#13

Post by geodel »

mjb wrote: post #9
geodel wrote: post #5

Good read. I like the part that both writing low level as well as high level component in Rust was proven worthwhile. Maybe one can transform slow code from high level languages to low level language via LLMs in future. That can be nice performance boost for those who don't have Amazon engineers and budgets

> Maybe one can transform slow code from high level languages to low level language via LLMs in future. This is one of the areas I'm most excited for LLM developer tooling. Choosing a language, database, or framework is a really expensive up-front decision for a lot of teams, made when they have the least information about what they're building, and very expensive to take back. If LLM-powered tools could take 10-100x…

I remember many years back one of Go language author wrote C to Go trasformer and used that to convert all compiler, runtime, GC etc into Go.

Now in today's time some experts like above could create base transformer for high level language and frameworks to low level language and frameworks and this all get exposed via llm interfaces.

One can say why all this instead of generating fast binary directly from high level code. But generating textual transformation would give developers opportunity to understand, tweak and adjust transformed code which generating direct binary would not.

Re: Just make it scale: An Aurora DSQL story

#14

Post by geodel »

karl_p wrote: post #10

The JVM can relocate memory to avoid fragmentation. Rust can't, at least natively. Are they not worried about this regression?

Well Java need it because it fragments memory a lot. With Rust one has value types and stack allocation which takes care of one of the biggest cause of fragmentation.

Re: Just make it scale: An Aurora DSQL story

#15

Post by bee_rider »

geodel wrote: post #5

Good read. I like the part that both writing low level as well as high level component in Rust was proven worthwhile. Maybe one can transform slow code from high level languages to low level language via LLMs in future. That can be nice performance boost for those who don't have Amazon engineers and budgets

Python -> C -> Assembly

Probably looks a lot like

Pseudocode -> C -> Assembly

Although the first is easier to run tests on and compare the outputs.

Re: Just make it scale: An Aurora DSQL story

#16

Post by Demiurge »

anarazel wrote: post #12
Demiurge wrote: post #8

Many interesting things, for instance, I've been hearing a lot about how fast Java is, that it can be as fast as C++, and then I see this: > But after a few weeks, it compiled and the results surprised us. The code was 10x faster than our carefully tuned Kotlin implementation – despite no attempt to make it faster. To put this in perspective, we had spent years incrementally improving the Kotlin version from 2,000 to…

> So PostgreSQL evolved to the point that it has a stable API for extensibility? Not across major versions, no. I seriously doubt we will ever make promises around that. It would hamper development way too much.

I see, then they're probably saying they found the internal APIs that are just more naturally stable, perhaps because they are close to the APIs used for extensions.

Re: Just make it scale: An Aurora DSQL story

#17

Post by loevborg »

glzone1 wrote: post #2

Early dsql had some weird limits I think - anyone actually using in production with feedback on current corners and limits?

Which ones? It seems eminently usable from the outside now, at least for greenfield work. The subset of Postgres it supports is most of good/core/essential Postgres. (But I haven't tried it)

Re: Just make it scale: An Aurora DSQL story

#18

Post by kikimora »

geodel wrote: post #14
karl_p wrote: post #10

The JVM can relocate memory to avoid fragmentation. Rust can't, at least natively. Are they not worried about this regression?

Well Java need it because it fragments memory a lot. With Rust one has value types and stack allocation which takes care of one of the biggest cause of fragmentation.

Writing code that would not fragment memory over time is arguable much harder than writing GC friendly code.

Re: Just make it scale: An Aurora DSQL story

#19

Post by avereveard »

mjb wrote: post #6
glzone1 wrote: post #2

Early dsql had some weird limits I think - anyone actually using in production with feedback on current corners and limits?

Which features would you like to see the team build first? Which limits would you like to see lifted first? Most of the limitations you can see in the documentation are things we haven't gotten to building yet, and it's super helpful to know what folks need so we can prioritize the backlog.

indexes! vector, trigram and maybe geospatial. (some may be in by now I didn't follow the service as closely as others)

note, doesn't have to be pg_vector pg_trgm or PostGIS, just the index component even if it's a clean room implementation would make this way more useful.

Re: Just make it scale: An Aurora DSQL story

#20

Post by geodel »

kikimora wrote: post #18
geodel wrote: post #14

Earlier quoted context omitted.

Well Java need it because it fragments memory a lot. With Rust one has value types and stack allocation which takes care of one of the biggest cause of fragmentation.

Writing code that would not fragment memory over time is arguable much harder than writing GC friendly code.

Yeah, cooking food in kitchen is much harder than having it delivered from restaurant at doorstep.

Reasonable people will see if cost makes it worthwhile.

Post reply on HN