[flagged]
Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
161–167 of 167 posts
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#162Earlier quoted context omitted.
Fuzzers are brilliant at this and produce all kinds of insane floating point inputs.
If you're only taking one float as input, we've long been at a point where you can just test them all: https://randomascii.wordpress.com/2014/01/27/theres-only-fou...
TDD haters in shambles with this news
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#163Earlier quoted context omitted.
My understanding is that they used c2rust, and then told the LLM/Agents to make the code more idiomatic rust, while also using the PG test suite as a feedback mechanism. This is almost certainly a derived work (and thus a fork, and should thus have the original license and copyright preserved). For example, if I compiled PG into x86-64 assembly, and then decompiled it into C (via, say, IDA), and then polished that de…
> My understanding is that they used c2rust, and then told the LLM/Agents to make the code more idiomatic rust, while also using the PG test suite as a feedback mechanism. This is correct > and thus a fork, and should thus have the original license and copyright preserved) This is not correct. The Postgres license is permissive. We need to include a copy of the license (which we do in the NOTICE file[0]) but we CAN r…
This feels very much like a distinction between letter of the law and spirit of the law. I knew a guy who once forked a very popular browser extension (you've definitely heard of it), that was copyleft-licensed. When I asked him where the repo is, he said it was not publicly available, but that the lawyers have assured him that he is not breaking any laws because the source code is available, via the inspect-extension feature of the browser. This answer has always felt problematic, to me.
In part, I feel like these legally-correct strategies and tactics tend be corrosive. In fact, this is exactly the kind of thing that drives projects like Sentry and Mongo to dump open source, and use EULAs instead. Not only is pgrust a license-based fork (which always runs the risk of splitting the community) it is also a language-based fork (which also runs the risk of splitting the community). In fact, the only silver lining here, is that you went straight for the AGPL, instead of MIT (thus preventing a _second_ license based fork).
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#164I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this? We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a pro…
Faster is better than slower. Postgres is amazing. When I use Sqlite I miss many of its upsides--better data migrations and better concurrency being two. But queries are not free, and sometimes are slower than I'd like even with the right indexes. And for OLAP-like queries like the one in TFA you might say "well, just don't use postgres for that," but wouldn't it be better if you could? If you could use one database…
So why not take the approach all the fast olap systems are going and serve as the indexing and query layer and put storage at the object store level?
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#165Reading about the batch optimization, I suspect there'll be trouble ahead for implementing window functions, unless those details were dropped for simplicity.
How so? Window functions work exactly the same way. Postgres processes them one row at a time, and you can batch them the same way as you would with sum. Window functions do make parallel queries more difficult, but that's a different story.
I guess regardless of if the calculations are batched or not, all the rows would have to be processed before the window function results could be determined.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#166Earlier quoted context omitted.
Interesting. It would need some work to handle all the workloads I’ve faced where you have two applications with different priorities (ie oltp or grid workloads and analytics). In those cases you want to make sure that BI users do not interrupt the transaction processing by assigning them a set of cores, a priority, and work mem limits for example. Looks doable without major changes.
We don't expose the priorities right now, but we have the priorities decay over time. That way faster queries get prioritized over long running queries. That should achieve the behavior you're looking for.
Yes for this particular use case it might help (no per-application aggregated temp file limit though). Also my grid workloads generate thousands of threads running queries with lots of small updates (job queues, calculation results..). I wish there was a way for postgresql to isolate those connections to a limited set of cores (or number of cores).
To avoid completely saturating the server.
Right now we advise using s3 and storing results in files, and asynchronously loading them with batch processes.
Re: Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD
#167Earlier quoted context omitted.
There is no “we” built it. You’re cosplaying as someone who could build a database. In reality you yourself could never build Postgres or Redis or any other database. You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres. Why would anyone ever trust you or this project?
> You’re cosplaying as someone who could build a database. > In reality you yourself could never build Postgres or Redis or any other database. > You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres. What makes you think these things? I've been writing about the Postgres internals and giving talks about Postgres for a decade and have managed…
Writing about Postgres internals and managing a Postgres cluster is the extent of your database expertise? You are not self aware enough to know that doesn’t earn any trust?