Using SIMD to aggregate billions of values per second
1–10 of 74 posts
Re: Using SIMD to aggregate billions of values per second
#2QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL.
High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages.
We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel execution to achieve performance figures at the top of the high-performance databases. Results are shown in our blog post.
I sincerely hope that you will find this helpful, and that this will unlock new possibilities! In the meantime, we will carry on working on QuestDB, adding more features, and taking this speed to more query types. Feel free to join us on slack or Github if you want to participate.
Re: Using SIMD to aggregate billions of values per second
#3Great work!
2.3x faster than kdb and 500x than postgres for sum (double) time series
What are your goals for the next release?
Re: Using SIMD to aggregate billions of values per second
#4I came across QuestDB in the past, but never tried myself. At my company, we use kx and onetick. Could you please elaborate why you are also comparing with Postgres since it's not really a time-series database nor revendicating to be part of the "high performance" club?
Re: Using SIMD to aggregate billions of values per second
#5The numbers are impressive, especially because it is against kdb. q/kdb is mostly finance focused and closed source so not really flexible. questdb has an advantage on this, it might be a bit of a tangent but I wonder if this could be used to replace redis, I can see how having SQL as a querying language could be a big plus.
Re: Using SIMD to aggregate billions of values per second
#6I came across QuestDB in the past, but never tried myself. At my company, we use kx and onetick. Could you please elaborate why you are also comparing with Postgres since it's not really a time-series database nor revendicating to be part of the "high performance" club?
You are right, PostgreSQL is not necessarily optimal for time-series workloads. In this case, the benchmark is a simple task, and not related to time-series. It's consists of reading 1 billion values to a table and sum them together. It doesn't get simpler than that.
The reason we are showcasing this instead of other more complex queries is because this is a simple, easily reproducible benchmark. It provides point of reference for performance figures.
Re: Using SIMD to aggregate billions of values per second
#7Great work! 2.3x faster than kdb and 500x than postgres for sum (double) time series What are your goals for the next release?
thank you! Next release will feature similar acceleration of group-by and time sampling queries.
Re: Using SIMD to aggregate billions of values per second
#8The numbers are impressive, especially because it is against kdb. q/kdb is mostly finance focused and closed source so not really flexible. questdb has an advantage on this, it might be a bit of a tangent but I wonder if this could be used to replace redis, I can see how having SQL as a querying language could be a big plus.
it would depend on the use case of course. QuestDB goes a bit further in that it provides speed close to in-memory loads while actually storing the data on disk. If there is enough interest, we could consider releasing an "in-memory only" mode for these use cases.
Re: Using SIMD to aggregate billions of values per second
#9So how does it compare to Clickhouse?
Re: Using SIMD to aggregate billions of values per second
#10So how does it compare to Clickhouse?
We have not benched against clickhouse yet. Sorry. Sounds like this should be an interesting things to do!