[deleted]
Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
131–140 of 167 posts
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#132Earlier quoted context omitted.
> I am very disappointed to see the direction: It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo" What makes you say this is a useless demo? I can't count the number of people who've struggled to do analytics inside of Postgres. Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they c…
I just deployed this to production to replace our self-hosted Postgres - and we’re experiencing corruption. Can you help us out?
Yes. Email me and Jason at malis@pgrust.com and jason@pgrust.com
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#133Earlier quoted context omitted.
I just deployed this to production to replace our self-hosted Postgres - and we’re experiencing corruption. Can you help us out?
Yes. Email me and Jason at malis@pgrust.com and jason@pgrust.com
Wow ok fair enough. Sorry to put you through a fire drill but I wanted to see if there would be support or not.
I stand corrected, there is. Wishing you the best of luck here.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#134I don't understand the use case. I've never seen why Postgres is better than anything else if I'm honest. Even if it is much, much faster, database transactions aren't what slow things down if written well in web applications.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#135I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this? We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a pro…
Faster is better than slower.
Postgres is amazing. When I use Sqlite I miss many of its upsides--better data migrations and better concurrency being two. But queries are not free, and sometimes are slower than I'd like even with the right indexes. And for OLAP-like queries like the one in TFA you might say "well, just don't use postgres for that," but wouldn't it be better if you could? If you could use one database instead of two, or use your favourite database in more places instead of reaching for a different tool?
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#136Earlier quoted context omitted.
> which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available That's up to GCP then. I'm sure OP would be willing to license pgrust to them for an appropriate price.
What's the incentive for third parties to contribute, knowing just one guy stands to benefit if the project gets monetized in such a manner?
Most of the same incentives of typical OSS contribution? It's not like other OSS projects pay out any earnings made with the product evenly across contributors.
Also, from the trajectory of the project it doesn't look like they are significantly deoendent on outside code contributions.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#137Earlier quoted context omitted.
If you are at that, add a native TTL feature as well. I think it’s been requested for decades.
What do you mean by native TTL? Would that be when rows are automatically deleted if they aren't touched after a certain period of time?
I'm guessing it's something like MongoDB TTL index[0]. It's useful for huge append-only tables where you want a row to be automatically removed after a period of time. The postgres approach is usually to partition your table by the event/insertion timestamp, and use something like pg_partman[1] to automatically drop entire partitions over time.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#138Earlier quoted context omitted.
People will use it if it really delivers gain and can solve problems PG can't.
It takes years to test it thoroughly in real environments. You don't want your data to be silently corrupted.
People have rapidly adopted far less tested databases when the dbs have claimed to solve real problems.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#139I am very disappointed to see the direction: It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo" Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
> I am very disappointed to see the direction: It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo" What makes you say this is a useless demo? I can't count the number of people who've struggled to do analytics inside of Postgres. Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they c…
> Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they can have one system that's Postgres-compatible, and it's faster than either of the original systems.
I can see the appeal for pgrust for smaller teams who need analytics, and don't want to deal with having to ETL to something like ClickHouse. But beyond a certain complexity, replicating your data to a warehouse or lakehouse _is_ the right approach and more scalable for several reasons:
- Analytics tend to be centralized, i.e. you want data from several Postgres databases spread across multiple teams to be replicated into 1 place, so people can start joining data across the entire business
- Analytics tend to fall under a different team ownership with their own set of non-technical requirements (e.g. data governance)
- Lakehouse architecture (Iceberg + [insert query engine]) is more scalable in terms of cost
- In some cases, you want to be able to swap different query engines depending on the use case, e.g. use PuppyGraph to query your data in Iceberg for fraud analysis
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#140Earlier quoted context omitted.
There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available. I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prev…
The whole point is to force the cloud providers to go through a contract negotiation for licensing rather than keep the door open for them. Nothing prevents the parent project from specially licensing to individual parties.
The issue is that AGPL is flatly banned at many companies, so now any company that even wants to test this database out is going to have to seek a contract, which they won't.