Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

Post reply on HN
Print view View original

121–130 of 167 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#121

Post by postgresperf »

btown wrote: post #18
malisper wrote: post #11

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

If someone wanted to use this as a real-time WAL-tracking read-only mirror of a live production database, for analytics work, is it ready for that use case yet?

When I got early access to pgrust my first question was whether it had been tested for building read-only standby analytics questions. The problem has goals beyond that, but having a real analytics node available just by playing WAL data from your primary makes pgrust a low risk eval for your data. Just have to build confidence the results are connection, which is being applied by all the PG infrastructure regression tests the code has been trained on,

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#122

Post by malisper »

wkoszek wrote: post #113
malisper wrote: post #11

Author here. Let me know if you have any questions about the post or about pgrust. Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust…

I really hope your project succeeds so that you can sit, sip delicious coffee with your legs on a high desk, and laugh really loud, like a gorilla, at all the haters from this and previous HN threads.

Thanks!

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#123

Post by ozgrakkurt »

sgt wrote: post #53

Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.

People are very willing to use new things when something they are using can't do what they want to do.

Dogma is more effective than reliability concerns in my experience. It just happens that the dogma prefers Postgres in this case.

It is just difficult to make these decisions in a more informed way because it requires actual manual testing with the concrete use case and also a decent understanding of databases in question. Chances are you won't have people that understand these things or those people won't have much of a say in decisions.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#124

Post by abdullahk0634 »

malisper wrote: post #10
cognitiveinline wrote: post #2

pgrust seems to have good momentum. AGPL is an odd license for a non web project. Postgres is MIT-like, and that drove it's adoption. Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.

Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard. For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackagi…

[dead]

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#125

Post by [deleted] »

wkoszek wrote: post #117

I'm really happy seeing this project. Not sure if this helps you gain $$$ customers, but stupid thing that turns out very difficult in PG is making this fast: SELECT COUNT(*) FROM large_text_db WHERE X Where X is something that must be matched exactly. X can be FTS query on FTS-indexed table, but the way COUNT() works in PG is that it's impossible to make it fast. Over large tables, lets say 1B+ rows, it can be very…

[deleted]

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#126

Post by hmokiguess »

I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this?

We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a problem it doesn’t fit?

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#128

Post by timacles »

malisper wrote: post #35
andriy_koval wrote: post #32

Earlier quoted context omitted.

what is your vision of this project? Do you think pgrust will eventually be prod ready?

I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.

What is your motivation? What you are doing is very impressive for sure, but the longer term life of this project seems incertain to me. While you might be able to accomplish your goal of production readyness, you'll only have version 1.0 at that point.

Its unlikely an AI written project is one which people will want to maintain even if it "works", which means AI has to maintain it. Maintaining a code base of a production database is going to continue costing 100s of thousands, if its even possible at all with AI. Its also unlikely to find users even if its almost perfect, because a bug in a database is not something users can accept.

Postgres is postgres because its like 30 years old and battle tested. Sure its missing some modern features, but its incredibly stable, which is part of its popularity.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#129

Post by malisper »

timacles wrote: post #128
malisper wrote: post #35

Earlier quoted context omitted.

I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.

What is your motivation? What you are doing is very impressive for sure, but the longer term life of this project seems incertain to me. While you might be able to accomplish your goal of production readyness, you'll only have version 1.0 at that point. Its unlikely an AI written project is one which people will want to maintain even if it "works", which means AI has to maintain it. Maintaining a code base of a produ…

I'm certain pgrust can find a long term home somewhere

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#130

Post by nullpoint420 »

malisper wrote: post #101
sunzhousz wrote: post #98

I am very disappointed to see the direction: It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo" Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.

> I am very disappointed to see the direction: It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo" What makes you say this is a useless demo? I can't count the number of people who've struggled to do analytics inside of Postgres. Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they c…

I just deployed this to production to replace our self-hosted Postgres - and we’re experiencing corruption. Can you help us out?
Post reply on HN