So how does it compare to Clickhouse?
We have not benched against clickhouse yet. Sorry. Sounds like this should be an interesting things to do!
The broader question though is how does your offering compare to Clickhouse?
Using SIMD to aggregate billions of values per second
51–60 of 74 posts
Re: Using SIMD to aggregate billions of values per second
#52I assume the values must be all be in memory beforehand and not hard storage.
That sum() SQL is all you do as a user. There is no additional magic there. The performance numbers are indeed best when data is in memory. However in reality sum() goes over memory mapped file, lazy loading data as required.
So how would this compare to the naïve approach of using a cursor in another fast, memory-mapped database, like LMDB?
Re: Using SIMD to aggregate billions of values per second
#53Earlier quoted context omitted.
Good summary, thank you. - we will extend SIMD to where clause, keyed aggregations, sampling, ordering, joins etc. It is a matter of time. - do you mind elaborating on how we screwed up date and time? - what makes you think we are never going to compete on performance with kdb+?
- nano are important for keeping ordering. while you may return results in insert order it is nice to have them so any other operations done in them outside the db you can retain (or recreate) that ordering. in financial systems nanos have become a sort of defacto standard for this. for examsple, all our messaging timestamps at places i've worked are always nano for anything written in the last 5-10 years. Also when…
I'm a QuestDB dev, with regards to nano timestamps, we dont use a nanosecond timestamp because its not possible to be accurate to that resolution with current hardware. However, on a single host the nano second clocks are precise and monotonic, they would be useful to maintain order. I think they do make sense and we will have to look into providing timestamps to that resolution.
Re: Using SIMD to aggregate billions of values per second
#54Earlier quoted context omitted.
Good summary, thank you. - we will extend SIMD to where clause, keyed aggregations, sampling, ordering, joins etc. It is a matter of time. - do you mind elaborating on how we screwed up date and time? - what makes you think we are never going to compete on performance with kdb+?
The reason why you cannot ever compete with kdb+/q is because the database and language run in one address space. Your benchmark gets around this problem by using the built in sum() function, but kdb+/q can just execute arbitrary code and never suffer a performance penalty. Unless you plan on integrated a high performance programming language into your DB, it simply will not be possible to ever meaningfully compete o…
I'm a QuestDB dev, data on a QuestDb is also stored in a single address space and SQL queries are compiled into objects that run in that address space. However, it is just SQL not a bespoke language. A future possibility would be to allow queries in java or scala.
Re: Using SIMD to aggregate billions of values per second
#55QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
Serious but bold question: What are the benefits versus Clickhouse for example? Why should I use QuestDB?
Amazing work either way. The space of databases can never have too much competition.
Re: Using SIMD to aggregate billions of values per second
#56Were these benchmarks before or after 2020.03.26? There was a bug that caused max operations to take twice as long.
From the KDB+ 4.0 release notes:
2020.03.26 FIX fixed performance regression for max. e.g. q)x:100000000?100;system"ts:10 max x"
Re: Using SIMD to aggregate billions of values per second
#57Were these benchmarks before or after 2020.03.26? There was a bug that caused max operations to take twice as long. From the KDB+ 4.0 release notes: 2020.03.26 FIX fixed performance regression for max. e.g. q)x:100000000?100;system"ts:10 max x"
these are against the latest 4.0 KDB+, after 26 March. KDB before that could not aggregate in parallel implicitly.
Re: Using SIMD to aggregate billions of values per second
#58QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
Just wanted to point out a small typo in the docs: "LASTEST BY" ( https://www.questdb.io/docs/crudOperations )
thanks a lot! fixed
Re: Using SIMD to aggregate billions of values per second
#59Earlier quoted context omitted.
We have not benched against clickhouse yet. Sorry. Sounds like this should be an interesting things to do!
The broader question though is how does your offering compare to Clickhouse?
co-founder of questdb here - we have been asked the same question on reddit as well. We are starting to work on an article going through a comparison between QuestDB and Clickhouse today - this will also include a bench. Will share as soon as we can. stay tuned!
Re: Using SIMD to aggregate billions of values per second
#60QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
I'll have to be careful here as I've no experience either with your DB or Postgres, but comparing questDB with PG and using that to claim it's 100X faster may be technically true but a bit of a dirty trick. It's perhaps like comparing a lorry with an F1 car and being surprised at the winner.
I'm also a little surprised at the speedup using SIMD. If it's RAM constrained, and it clearly is because it scales with the number of memory channels, then I completely fail to understand how instructions - which are bloody fast to execute (SIMD or not) compared to slow RAM access - could benefit you. Both SIMD or not should be waiting on memory AFAICS. Except you clearly do get a big speedup, could anyone enlighten me on this?