Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

Post reply on HN
Print view View original

171–180 of 182 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#171

Post by repsilat »

a34729t wrote: post #164
repsilat wrote: post #135

Earlier quoted context omitted.

Faster is better than slower. Postgres is amazing. When I use Sqlite I miss many of its upsides--better data migrations and better concurrency being two. But queries are not free, and sometimes are slower than I'd like even with the right indexes. And for OLAP-like queries like the one in TFA you might say "well, just don't use postgres for that," but wouldn't it be better if you could? If you could use one database…

So why not take the approach all the fast olap systems are going and serve as the indexing and query layer and put storage at the object store level?

Maybe that's the right approach, hard to say. No great harm making this kind of improvement if it isn't the globally best architectural direction.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#173

Post by hmokiguess »

repsilat wrote: post #135
hmokiguess wrote: post #126

I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this? We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a pro…

Faster is better than slower. Postgres is amazing. When I use Sqlite I miss many of its upsides--better data migrations and better concurrency being two. But queries are not free, and sometimes are slower than I'd like even with the right indexes. And for OLAP-like queries like the one in TFA you might say "well, just don't use postgres for that," but wouldn't it be better if you could? If you could use one database…

> wouldn't it be better if you could? If you could use one database instead of two, or use your favourite database in more places instead of reaching for a different tool?

No, I used to but not anymore. I do appreciate when solving the problem is enjoyable with a tool that brings in a good experience, but I try not to get attached these days.

I get what you're saying though.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#174

Post by hmokiguess »

mhh__ wrote: post #156
hmokiguess wrote: post #126

I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this? We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a pro…

There should be a halfway point between kdb and postgres... So more seriously my thinking is that we should, on purely moral grounds strive to have things be fast and lean. How many years of human misery have been lost to waiting around for python programs for example because the team for years wanted to the interpreter to be simple

I find your phrase contradicting in itself. You can't have your cake and eat it too. Where do you draw the line? Fast and lean means killing abstractions and DX. Look at Taalas burning the model weights into the wafer for example.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#175

Post by touisteur »

insanitybit wrote: post #138
f311a wrote: post #90

Earlier quoted context omitted.

It takes years to test it thoroughly in real environments. You don't want your data to be silently corrupted.

People have rapidly adopted far less tested databases when the dbs have claimed to solve real problems.

Especially if it can be run as a read-only replica.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#176

Post by ForHackernews »

insanitybit wrote: post #141
ForHackernews wrote: post #40

Earlier quoted context omitted.

> It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it? Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?

It's not that companies won't pay for it, it's that it is banned . Legal teams at these companies set the policy.

The fact that they ban themselves from using the free AGPL license is what makes it possible to sell them a commercial one.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#177

Post by insanitybit »

ForHackernews wrote: post #176
insanitybit wrote: post #141

Earlier quoted context omitted.

It's not that companies won't pay for it, it's that it is banned . Legal teams at these companies set the policy.

The fact that they ban themselves from using the free AGPL license is what makes it possible to sell them a commercial one.

Not really. How would they even evaluate the software? This only works in theory.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#178

Post by theplumber »

malisper wrote: post #112
theplumber wrote: post #110

Earlier quoted context omitted.

If you are at that, add a native TTL feature as well. I think it’s been requested for decades.

What do you mean by native TTL? Would that be when rows are automatically deleted if they aren't touched after a certain period of time?

Yes, TTL like on redis for example

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#179

Post by ForHackernews »

hunterpayne wrote: post #36
xyzzy_plugh wrote: post #23

Earlier quoted context omitted.

They could simply spend a few months and a few million tokens and get their own port, no? I doubt even 30000x faster would prompt a policy change.

No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. In all fairness, your average app programmer would have the same problems. That's why it takes so long to learn to be a system programmer and why it takes so long to do anything on a systems source base. For reference, systems are OSes, DBs and compilers (although compilers…

You're screaming into the wind. LLMs are coming for "system code" along with everything else. It no longer takes time to learn to be a systems programmer, or time to do things on a systems codebase. You can complain about the bugs the bots introduce but no one will care, least of all the bosses who are sick of paying for slow, expensive system programmers.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#180

Post by ForHackernews »

insanitybit wrote: post #177
ForHackernews wrote: post #176

Earlier quoted context omitted.

The fact that they ban themselves from using the free AGPL license is what makes it possible to sell them a commercial one.

Not really. How would they even evaluate the software? This only works in theory.

Selling commercial database software to big enterprises seems to work in practice for Oracle.
Post reply on HN