411–420 of 756 posts
Earlier quoted context omitted.
Or, you know, you can use Postgres. It's right there for you.
why? if a rewrite is better/faster/secure, why not? (I'm not saying PGrust is better, I didnt even install it, my perspective is in general)
Earlier quoted context omitted.
can you enlighten me, what exactly do you learn from asking a llm to do a rewrite?
I mean you can learn a lot. You can learn what's possible with less effort to build a proof of concept. It's kind of like you had another engineer do it for you, you don't completely learn how to do it yourself but you can still learn a lot with much less effort
Earlier quoted context omitted.
> - Is ~300x faster than Postgres on analytical workloads. Right now it's 2x slower than Clickhouse on clickbench and I think it's possible to get faster than Clickhouse That sounds like you are storing the data in a columnar format? Or do you do both row and columnar? In a somewhat similar (yet also quite different) effort, I've been working on δx, a Postgres extension that compresses the data in a columnar format s…
Yep! The new version of pgrust supports batch based execution and a columnar format. I'm curious how you got δx to perform that well? From what I've seen a columnar layout only gets you part of the way and really good parallelism and really fast hash tables seem to make up a significant portion of why Clickhouse is faster.
pg_lake also uses DuckDB but keeps it external, routing through Postgres and managing Iceberg tables (but not the data itself) there (https://github.com/Snowflake-Labs/pg_lake).
Both of these were neck and neck with ClickHouse last time I tried them.
Earlier quoted context omitted.
While a thread-per-connection seems like an improvement, do you have any plans to allow query multiplexing over a single connection? That would be a huge improvement IMO.
Can you elaborate on the use case for query multiplexing? Is it so your client would only need to establish one connection with Postgres and then could run as many queries as it wanted?
On the client side, there is usually a local connection pool. When a burst of traffic comes in, the client needs to either wait for the pool to free up or establish a new connection, which adds latency. This latency hit wouldn’t occur with multiplexing.
With multiplexing, systems like pgbouncer would be unnecessary.
Also, even with a thread-per-connection, you can still quickly exhaust the servers resources when you have lots of connections because threads have a lot of overhead. Reducing the number of connections needed would greatly increase the number of clients that a database can serve.
Hey author here. Wasn't expecting to see this up. To concisely give an overview of the project, I've been experimenting with using LLMs to build a better version of Postgres. Postgres is 30 years old and we've learned a lot about databases since hten. A lot of the techniques that work for doing a rewrite are also useful for doing a rearchitecture. I'm now working on a new, not yet published version of pgrust that inc…
A thread per connection is a almost always the correct decision for performance, but by choosing a process per connection, postgres is able to let you load whatever sketchy extensions you want. Worst case you crash the process, not the database. It would be nice if you could strike a balance so a segfaul in the extension only crashes a small percentage of connections, not the whole thing.
Clearly a process per connection is more stable and that's what I'm using.
It's unclear what problem such optimizations are solving anyway, with the old way you could only support a million concurrent users with a single server? Are we missing out on supporting ten million concurrent users with 2 servers instead of 10? Ostensibly reducing the minimum db hardware opex for a 10B$ company from 10k$/month to 2k$/month?
Don’t understand these rewrites. - typically they are behind a single person. That’s usually bad because of spf - typically they are achieved in a very short amount of time, so the author hasn’t acquired any discipline in creating the project. That means it’s unlikely the author is going to stick to the project in the mid and long term - anyone that wants to contribute to the project needs to pay. Needs to pay tokens…
I likely wouldn't use a rewrite of such a huge project if it doesn't have the backing of the original team (or a significant fraction thereof) and a believable story for having matched/exceeded the original code quality and maintenance. I also think in general using an LLM for license-laundering is legally and morally hard to defend, although this case is different in that they chose a more restrictive license. Not a lawyer, but my understanding is that you can just download PostgreSQL, do s/MIT/AGPL/ and release it, legally. (The original MIT-licensed version still exists, so no reason anyone would prefer yours until you make another release with some compelling new feature.)
I have a feeling that AI is rewriting everything!
First bun, now it's PG's turn again, although this isn't official. I have a feeling that AI is rewriting everything!