Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

450× Faster Joins with Index Condition Pushdown

readyset.io

Post reply on HN
Print view View original

51–60 of 61 posts

Re: 450× Faster Joins with Index Condition Pushdown

#51

Post by OutOfHere »

airstrike wrote: post #44
OutOfHere wrote: post #40

Earlier quoted context omitted.

What you had claimed is not even a potential reason in the universe of reasons. It is a demonstration of bias, an excuse to refrain from reason. One line summaries of comprehensible articles can get downvoted because they don't add value beyond what's already very clear from the article.

it is objectively a potential reason in the universe of reasons, but you're 100% free to believe whatever you want, even if it's wrong and the fact that multiple people upvoted my comment at a minimum suggests others also believe it to be a possible explanation i have no idea why you've chosen this particular hill to die on, when neither of us stands to profit from this protracted exchange

What happens is that some people routinely use your purported reason "it's LLM generated" as an excuse to try to discredit anything at all, and it's not right, irrespective of whether the material is LLM generated or not. Any material should be critiqued on the basis of its own merits and demerits, irrespective of who or what authored it. We need to shed the pro-human bias.

Re: 450× Faster Joins with Index Condition Pushdown

#52

Post by airstrike »

OutOfHere wrote: post #51
airstrike wrote: post #44

Earlier quoted context omitted.

it is objectively a potential reason in the universe of reasons, but you're 100% free to believe whatever you want, even if it's wrong and the fact that multiple people upvoted my comment at a minimum suggests others also believe it to be a possible explanation i have no idea why you've chosen this particular hill to die on, when neither of us stands to profit from this protracted exchange

What happens is that some people routinely use your purported reason "it's LLM generated" as an excuse to try to discredit anything at all, and it's not right, irrespective of whether the material is LLM generated or not. Any material should be critiqued on the basis of its own merits and demerits, irrespective of who or what authored it. We need to shed the pro-human bias.

Hard disagree. In fact, I'm very much pro-human and anti-unqualified "we need to..." statements

Either way, I didn't even downvote the OP so you're beeping at the wrong human

Re: 450× Faster Joins with Index Condition Pushdown

#53

Post by OutOfHere »

airstrike wrote: post #52
OutOfHere wrote: post #51

Earlier quoted context omitted.

What happens is that some people routinely use your purported reason "it's LLM generated" as an excuse to try to discredit anything at all, and it's not right, irrespective of whether the material is LLM generated or not. Any material should be critiqued on the basis of its own merits and demerits, irrespective of who or what authored it. We need to shed the pro-human bias.

Hard disagree. In fact, I'm very much pro-human and anti-unqualified "we need to..." statements Either way, I didn't even downvote the OP so you're beeping at the wrong human

I am pro-truth. Being pro-truth is more pro-human in the long term via indirect effect, than is being pro-human directly. Focusing on being pro-human can reward bad behavior among masses of humans, leading to their ultimate downfall. I will leave it at that.

Re: 450× Faster Joins with Index Condition Pushdown

#55

Post by pragmatic »

dinobones wrote: post #41
futevolei wrote: post #39

Do the db guys at your company help you optimize queries and table set up at all? Ours basically don’t at all. Their job is to maintain the db apparently and us devs are left to handle this and it seems wrong. I’ve been partitioning tables and creating indexes the past few weeks trying to speed up a view and running explain analyze and throwing the results in Gemini and my queries are still slow af. I had one sql cla…

I always see these fancy DB engines and data lake blog posts and I am curious… why? At every place I’ve worked at this is a solved problem: Hive+Spark, just keep everything sharded across a ton of machines. It’s cheaper to pay for a Hive cluster that does dumb queries than paying these expensive DB licenses, data engineers building arbitrary indices, etc… just throw compute at the problem, who cares. 1TB of RAM /flas…

A fellow data swamp enjoyer.

Just dump it in Hadoop became an anti-pattern and everyone yearned for databases and clean data and not dealing with internal IT and the cluster “admins”.

Re: 450× Faster Joins with Index Condition Pushdown

#56

Post by pragmatic »

aseipp wrote: post #22
flufluflufluffy wrote: post #16

I read their website landing page but it’s still kinda confusing — what exactly is readyset? It all sounds like it’s a cache you can set up in front of MySQL/postgres. But then this article is talking about implementing joins which is what the database itself would do, not a cache. But then the blurbs talk about it like it’s a “CDN for your database” that brings your data to the edge. What the heck is it?!

ReadySet is basically "incremental view maintenance" but applied to arbitrary SQL queries. It acts like a caching proxy for your database, but it simultaneously ingests the replication log from the system in order to see things happen. Then it uses that information to perform "incremental" updates of data it has cached, so that if you requery something, it is much faster. Naive example: let's say you had a query that…

Indexed Views for Postgres but at the query level?

Re: 450× Faster Joins with Index Condition Pushdown

#57

Post by pragmatic »

futevolei wrote: post #39

Do the db guys at your company help you optimize queries and table set up at all? Ours basically don’t at all. Their job is to maintain the db apparently and us devs are left to handle this and it seems wrong. I’ve been partitioning tables and creating indexes the past few weeks trying to speed up a view and running explain analyze and throwing the results in Gemini and my queries are still slow af. I had one sql cla…

95% it’s a missing/improper index but the insane workarounds “engineers” put in place actively work against the database.

Does your hotpath query have a covering index? Is it seek-able. Are you predicates SARG-able.

Seriously biggest bang for your buck is to understand SARG-ability and understand indexes in your database really well.

Re: 450× Faster Joins with Index Condition Pushdown

#58

Post by pragmatic »

futevolei wrote: post #39

Do the db guys at your company help you optimize queries and table set up at all? Ours basically don’t at all. Their job is to maintain the db apparently and us devs are left to handle this and it seems wrong. I’ve been partitioning tables and creating indexes the past few weeks trying to speed up a view and running explain analyze and throwing the results in Gemini and my queries are still slow af. I had one sql cla…

What are “db guys” in this context?

There are two distinct job functions.

1) dba - maintain OS, db software/hardware - DBs are complex beasts you want experts setting up hardware or deploying into the clown in a smarter way (virtual machines in the clown will cost you a fortune and your sanity) 2) database developers- specialists in writing sql.

The two functions work together but are distinct.

Nowadays due to private equity and Stanford MBAs we have junior engineers doing all plus “dev ops”.

It’s an absolute circus.

IMO, this is why we end up with ask these crazy DB startups - routing around the damage.

RDMS in modern hardware are insanely fast and powerful.

Re: 450× Faster Joins with Index Condition Pushdown

#59

Post by jmalicki »

Seattle3503 wrote: post #50
jmalicki wrote: post #33

Earlier quoted context omitted.

Without storing the joint distribution of the values corresponding to the conditions that span multiple tables, it can be hard to know what's a win.

Postgres by default computes univariate stats for each column and uses those. If this is producing bad query plans, you can extend the statistics to be multivariate for select groups of columns manually. But to avoid combinatorially growth of stats related storage and work, you have to pick the columns by hand.

That still isn't multivariate stats across a join.

Re: 450× Faster Joins with Index Condition Pushdown

#60

Post by marceloaltmann »

zmmmmm wrote: post #43
HDThoreaun wrote: post #27

Earlier quoted context omitted.

Yea this is pretty fucking basic stuff. Any competent optimization engine should be doing this. "push down indexes as much as possible" is literally the first thing a query planner should be trying to do

I had to dig through to see the details of what database was really in play here, and sure enough, it's a wrapper around a key-value store (RocksDB). So while I'll confess I know little about RocksDB it does sound an awful lot like they threw out a mature relational database engine with built in optimization and now are in the process of paying the price for that by manually optimizing each query (against a key-value…

Readyset is an Incremental View Maintenance cache that is powered by a dataflow graph to keep caches (result-set) up-to-date as the underlining data changes on the database (MySQL/PostgreSQL). RocksDB is only used as the persistent storage here, and the whole optimization is done for the DFG execution and not related to the persistent storage itself.
Post reply on HN