Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No, QuestDB is not Faster than ClickHouse

telegra.ph

Post reply on HN
Print view View original

51–60 of 73 posts

Re: No, QuestDB is not Faster than ClickHouse

#51

Post by bluestreak »

PeterZaitsev wrote: post #36
bluestreak wrote: post #20

Earlier quoted context omitted.

I am in fact very proud of my team, who worked very hard on both implementation and the article. It is disappointing to read unfounded insults where we made every effort to be fair.

I wonder what "every effort to be fair" means ? The first thing you could have done is reach out to ClickHouse Community to ask for optimization suggestions

"fair" means that we comparing apples to apples. Ad-hoc, unindexed predicate, compiled by QuestDB into AVX2 assembly (using AsmJIT) vs same predicate complied by Clickhouse (I'm assuming by LLVM). One can perhaps view this as comparing SIMD-based scans from both databases. Perhaps we generate better assembly, which incidentally offers better IO.

We all understand that creating very specific index might improve specific query performance. Great, Clickhouse geared the entire table storage model to be ultra specific for latitude search. What if you search by longitude, or other column? Back to the beginning.

JIT-compiled predicates offer arbitrary query optimisation with zero impact on ingestion. This is sometimes useful.

What would you offer assuming that we reached out, other than creating an index?

Clickhouse does better than we do in other areas. It JITs more complicated expressions, such as some date functions. It optimises count() queries specifically. For example we collect "found" rowed_ids in an array. Clickhouse does not specifically for count(). We still have work to do. On other hand we ingested this very dataset about 5x quicker than clickhouse, which we left out because article is not about "QuestDB is faster than Clickhouse"

Re: No, QuestDB is not Faster than ClickHouse

#53

Post by untitaker_ »

PeterZaitsev wrote: post #35
bluestreak wrote: post #11

Our article in question can be found here: https://questdb.io/blog/2022/05/26/query-benchmark-questdb-v... The intent of the article was to showcase JIT-optimised WHERE clause and we did not use any indexes on QuestDB.

If your intent it to showcase the new optimization in the product it is best to compare it to your own old version

Right? How do the folks at QuestDB know that their new JIT engine is actually responsible for those performance improvements? My understanding is that, index or not, data is still sorted by time in questdb, which is exactly what the ClickHouse engineers are replicating in the new schema.

Re: No, QuestDB is not Faster than ClickHouse

#54

Post by gfody »

bombcar wrote: post #23
noxvilleza wrote: post #15

Earlier quoted context omitted.

Oh dear. I did a brief search for 'adage on benchmarking' and only saw Rugg/Feldman benchmarks.

It's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.

there are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp

you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/

Re: No, QuestDB is not Faster than ClickHouse

#55

Post by bluestreak »

untitaker_ wrote: post #53
PeterZaitsev wrote: post #35

Earlier quoted context omitted.

If your intent it to showcase the new optimization in the product it is best to compare it to your own old version

Right? How do the folks at QuestDB know that their new JIT engine is actually responsible for those performance improvements? My understanding is that, index or not, data is still sorted by time in questdb, which is exactly what the ClickHouse engineers are replicating in the new schema.

The query Clickhouse picked on does not actually leverage time order. Perhaps clickhouse vendors on this thread can comment on relevance of the date partitioning for this query. My best guess is that it might help the execution logic to create data chunks for parallel scan.

QuestDB does also use partitions for this purpose but we also calculate chunks dynamically based on available CPU to distribute load across cores more evenly

Re: No, QuestDB is not Faster than ClickHouse

#56

Post by hodgesrm »

bluestreak wrote: post #20
datalopers wrote: post #18

Earlier quoted context omitted.

Quoted post unavailable.

I am in fact very proud of my team, who worked very hard on both implementation and the article. It is disappointing to read unfounded insults where we made every effort to be fair.

I appreciate your benchmark and was interested to learn about how QuestDB processes TSBS queries efficiently. I work extensively with ClickHouse and it's always enlightening to learn about how other databases achieve high performance. Your descriptions of the internals are clear and easy to follow, especially since you included comparisons with older versions of QuestDB.

That said, I think I can understand how some users might be a little put off by the comparisons. Your article effectively says "ClickHouse is really slow" without giving readers any easy way to judge what was happening under the covers. I was personally a bit frustrated not to have the time to set up TSBS and dig into what was going on. I therefore appreciated Geoff's effort look up the results and show that the default index choices didn't make a lot of sense for this particular case. That does not detract from QuestDB's performance at least from my perspective.

Anyway congratulations on the performance improvement. As a famous character in Star Wars said, "we will watch your career with great interest."

edit: correct typo

Re: No, QuestDB is not Faster than ClickHouse

#57

Post by olluk »

PeterZaitsev wrote: post #35
bluestreak wrote: post #11

Our article in question can be found here: https://questdb.io/blog/2022/05/26/query-benchmark-questdb-v... The intent of the article was to showcase JIT-optimised WHERE clause and we did not use any indexes on QuestDB.

If your intent it to showcase the new optimization in the product it is best to compare it to your own old version

Comparison with old version is actually in the article for the patient reader. It could go to the top but I don't think it will make a difference. At the end of the day it is the article at the official QuestDB website which gives the reader a spoiler about the bias.

I am intrigued what Timescale is going to publish next.

Re: No, QuestDB is not Faster than ClickHouse

#58

Post by qoega »

gfody wrote: post #54
bombcar wrote: post #23

Earlier quoted context omitted.

It's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.

there are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/

It is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.

Re: No, QuestDB is not Faster than ClickHouse

#59

Post by ceejayoz »

gauravphoenix wrote: post #26

I have always felt that DB benchmarks are useless, always use your own dataset https://gauravkumar.blog/performance-benchmarks-are-useless....

That has the confounding variable of how good you are at configuring each database option.

I can configure Postgres fairly well. I have little chance of knowing if I’m getting good performance out of most others without a serious time investment.

Re: No, QuestDB is not Faster than ClickHouse

#60

Post by avianlyric »

dimgl wrote: post #38
avianlyric wrote: post #32

Earlier quoted context omitted.

What an extremely unfair comment. Having read QuestDBs blog, it’s quite clear they’ve taken great pains to point out that a single specific benchmark isn’t the be all and end all of DB analysis. They quite clearly start out by saying they’re only looking to demonstrate the impact of a specific new DB feature they’ve created, and are using benchmarks that illustrate the difference. They make zero claims that QuestDB i…

> They make zero claims that QuestDB is faster than Clickhouse overall Are you sure? Just one look at their website says differently. https://questdb.io/time-series-benchmark-suite/ I don't use these tools. I just wanted to point out that what you're saying is disingenuous.

I’m commenting specifically on the blog post provided by GP, which the parent comment made some pretty derogatory comments about.

I’ve made no attempt to deeply research what QuestDB have said else where, because I don’t care. I don’t use, or have a need, for any of the products mentioned in any of the linked articles. I’m only interested in the narrow discussion of the original blog post provided by GP, to which OP post is replying to.

Post reply on HN