Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

malisper.me

Post reply on HN
Print view View original

81–90 of 169 posts

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#81

Post by Tuna-Fish »

claytonjy wrote: post #68
wiradikusuma wrote: post #67

I think in addition to making it faster, it would be useful if it could be made "leaner," e.g. can run better on lower-spec hardware than PG.

does one not imply the other? if it can run faster in the same hardware, it should also run as fast on lower spec hardware

That is not a given. A database server can run faster on better hardware because it more effectively and aggressively caches things in memory, which can hurt it on lower-spec systems. Or it can better utilize SIMD instructions that are not present on the low end. Or it is more effective at utilizing more threads, but is slower when run at a low threadcount, etc etc etc.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#82

Post by smolder »

claytonjy wrote: post #68
wiradikusuma wrote: post #67

I think in addition to making it faster, it would be useful if it could be made "leaner," e.g. can run better on lower-spec hardware than PG.

does one not imply the other? if it can run faster in the same hardware, it should also run as fast on lower spec hardware

No, it doesn't. Different algorithms can vary pretty wildly in performance based on the design of the hardware they run on. For instance cache sizes can make one implementation of a sort on a certain sized dataset faster or slower than another. You can have a theoretically fast algorithm that just isn't as cache efficient as a theoretically slower one in big-O terms. All levels of the memory hierarchy as well as storage have specific bandwidths and latencies that inform the real world performance results. Parallelism is another issue. Many cores can do work very fast when you're careful about how you split up work between them, taking into consideration the synchronization latency and individual cache sizes and so on. The best approach for doing work on 64 cores can be dramatically different from what works best on 1 or even 2.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#83

Post by yieldcrv »

sgt wrote: post #80
yieldcrv wrote: post #71

Earlier quoted context omitted.

They will be software engineers or the people doing the software work And different people will be doing product management and HR all because today’s software engineers don’t want to babysit AI agents and choose antiquated libraries so that their resume said they'd been using a popular framework for a couple years Nobody else is playing that outdated game, its just a rotation

What are we going to do about this?

this thread is talking about using a 300x faster library and theorizing a resistance to devs using it

just use the faster library? leverage compute resources more effectively and justify value to an organization better than the next person

otherwise, what needs to be done? I don’t see a problem with any of this aside from organizations risking less experienced people doing less efficient things in other parts of the stack

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#84

Post by ZiiS »

Surly AI could also write a clearer headline. For the millions running it in production for decades, using a great echosystem of help support, books, consultants, and managed hosting providers; the is a noteworthy difference between the official release and a partially compatable rewrite.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#85

Post by booksock »

refulgentis wrote: post #76
malisper wrote: post #75

Earlier quoted context omitted.

We disabled parallelism in the blog post for demonstration purposes. The 300x slower refers to the clickbench numbers[0] where parallelism is enabled [0] https://benchmark.clickhouse.com/#system=+liH|pgrs|gQ&type=-...

Who is "we"?

hi

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#86

Post by shawnz »

hobofan wrote: post #58
mey wrote: post #14

Earlier quoted context omitted.

There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available. I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prev…

> which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available That's up to GCP then. I'm sure OP would be willing to license pgrust to them for an appropriate price.

What's the incentive for third parties to contribute, knowing just one guy stands to benefit if the project gets monetized in such a manner?

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#87

Post by hunterpayne »

skinfaxi wrote: post #52
hunterpayne wrote: post #36

Earlier quoted context omitted.

No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. In all fairness, your average app programmer would have the same problems. That's why it takes so long to learn to be a system programmer and why it takes so long to do anything on a systems source base. For reference, systems are OSes, DBs and compilers (although compilers…

> No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. What does it being system code have to do with anything?

The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this.

Most systems code has to be "canonical". That means it can't be improved. Imagine the most difficult PR review you have ever had. Now give the reviewers OCD and an extra 30 IQ points. And every single code change has to live up to this standard. That's what it has to do with this.

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#88

Post by imhoguy »

KolmogorovComp wrote: post #51

Thanks to the authors for choosing a license that respect users freedom, on top of being an awesome technical project.

Does that license protect anything legally? This is written by AI under some initial guidance. Prompts may be protected but the output of unlicensed knowledge model corpus?

Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

#90

Post by f311a »

andriy_koval wrote: post #54
sgt wrote: post #53

Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.

People will use it if it really delivers gain and can solve problems PG can't.

It takes years to test it thoroughly in real environments. You don't want your data to be silently corrupted.
Post reply on HN