Live data from Hacker News

450× Faster Joins with Index Condition Pushdown

readyset.io

11–20 of 61 posts

Re: 450× Faster Joins with Index Condition Pushdown

#11
Another example of row based dbs somehow being insanely slow compared to column based.

Just an endless sequence of misbehavior and we’re waving it off as rows work good for specific lookups but columns for aggregations, yet here it is all the other stuff that is unreasonably slow.

Re: 450× Faster Joins with Index Condition Pushdown

#12

Another example of row based dbs somehow being insanely slow compared to column based. Just an endless sequence of misbehavior and we’re waving it off as rows work good for specific lookups but columns for aggregations, yet here it is all the other stuff that is unreasonably slow.

It's an example. But not of that.

It's an example of old things being new again maybe. Or reinventing the wheel because the wheel wasn't known to them.

Yes I know, nobody wants to pay that tax or make that guy richer, but databases like Oracle have had JPPD for a long time. It's just something the database does and the optimizer chooses whether to do it or not depending on whether it's the best thing to do or not.

Re: 450× Faster Joins with Index Condition Pushdown

#13

Another example of row based dbs somehow being insanely slow compared to column based. Just an endless sequence of misbehavior and we’re waving it off as rows work good for specific lookups but columns for aggregations, yet here it is all the other stuff that is unreasonably slow.

I feel like this is more an example of:

“We filtered first instead of reading an entire table from disk and performing a lookup”

Where both OLAP and OLTP dbms would benefit.

To your point, it’s clear certain workloads lend themselves to OLAP and columnar storage much better, but “an endless sequence of misbehavior” seems a bit harsh .

Re: 450× Faster Joins with Index Condition Pushdown

#14

Straddled joins were still a bottleneck in Readyset even after switching to hash joins. By integrating Index Condition Pushdown into the execution path, we eliminated the inefficiency and achieved up to 450× speedups.

Why downvote?

Reads like an ad written by an LLM, is my guess.

It could just be that they translated from their original language to English and got that as a byproduct. Many such cases.

Re: 450× Faster Joins with Index Condition Pushdown

#15

Another example of row based dbs somehow being insanely slow compared to column based. Just an endless sequence of misbehavior and we’re waving it off as rows work good for specific lookups but columns for aggregations, yet here it is all the other stuff that is unreasonably slow.

It's an example. But not of that. It's an example of old things being new again maybe. Or reinventing the wheel because the wheel wasn't known to them. Yes I know, nobody wants to pay that tax or make that guy richer, but databases like Oracle have had JPPD for a long time. It's just something the database does and the optimizer chooses whether to do it or not depending on whether it's the best thing to do or not.

Exactly. This is a basic optimization technique and all the dinosaur era databases should have that. But if you build a new database product you have to implement these techniques from scratch. There is no way you shortcut that. Reminds me about CockroachDB and them building a query optimizer[1]. They started with rule based one and then switched to cost based. Feature that older databases already had.

[1] https://www.cockroachlabs.com/blog/building-cost-based-sql-o...

Re: 450× Faster Joins with Index Condition Pushdown

#16
I read their website landing page but it’s still kinda confusing — what exactly is readyset? It all sounds like it’s a cache you can set up in front of MySQL/postgres. But then this article is talking about implementing joins which is what the database itself would do, not a cache. But then the blurbs talk about it like it’s a “CDN for your database” that brings your data to the edge. What the heck is it?!

Re: 450× Faster Joins with Index Condition Pushdown

#17
Maybe it's not obvious initially, but in retrospect, this handling of joins feels like the obvious way to handle it.

Push down filters to read the least data possible.

Or, know your data and be able to tell the query engine which kind of join strategy you would like (hash vs push down)

Re: 450× Faster Joins with Index Condition Pushdown

#18

Maybe it's not obvious initially, but in retrospect, this handling of joins feels like the obvious way to handle it. Push down filters to read the least data possible. Or, know your data and be able to tell the query engine which kind of join strategy you would like (hash vs push down)

Decades ago we used to provide hints in queries based on "knowing the data" but modern optimizers have a lot better statistics on indexes, and the need to tell the query optimizer what to do should be rare.

Re: 450× Faster Joins with Index Condition Pushdown

#19

Maybe it's not obvious initially, but in retrospect, this handling of joins feels like the obvious way to handle it. Push down filters to read the least data possible. Or, know your data and be able to tell the query engine which kind of join strategy you would like (hash vs push down)

[deleted]

Re: 450× Faster Joins with Index Condition Pushdown

#20

I read their website landing page but it’s still kinda confusing — what exactly is readyset? It all sounds like it’s a cache you can set up in front of MySQL/postgres. But then this article is talking about implementing joins which is what the database itself would do, not a cache. But then the blurbs talk about it like it’s a “CDN for your database” that brings your data to the edge. What the heck is it?!

It seems to be some sort of read-only reimplementation of MySQL/Postgres that can ingest their replication streams and materialize views (for caching). Complete with a really primitive optimizer, if the article is to be believed.
Post reply on HN