What about data compression? How does that compare to other time series DBs?
Using SIMD to aggregate billions of values per second
11–20 of 74 posts
Re: Using SIMD to aggregate billions of values per second
#12What about data compression? How does that compare to other time series DBs?
There is no dedicated data compression at the moment as our use-case is centered around performance. That being said, we store data very efficiently with minimal overhead.
One interesting consequence is that in many cases, we don't require indexes where other databases do. This synthetically compress the data relative to other DBs by removing the space taken by these indexes while improving query speed. It also means ingestion speed remains fast with O(1) complexity.
Re: Using SIMD to aggregate billions of values per second
#13What about data compression? How does that compare to other time series DBs?
There is no dedicated data compression at the moment as our use-case is centered around performance. That being said, we store data very efficiently with minimal overhead. One interesting consequence is that in many cases, we don't require indexes where other databases do. This synthetically compress the data relative to other DBs by removing the space taken by these indexes while improving query speed. It also means…
> we store data very efficiently with minimal overhead.
Do you use encoding like delta-of-delta timestamps or something similar?
> One interesting consequence is that in many cases, we don't require indexes where other databases do.
I don't follow. Why don't you require indexes where other databases do?
Thanks.
Re: Using SIMD to aggregate billions of values per second
#14[deleted]
Re: Using SIMD to aggregate billions of values per second
#15QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
Thank you for advancing the state of the art!
Do you consider offloading some code to the gpu at some point?
JITs are really good at doing speculative optimization then deoptimizing if the optimization failed. Couldn't DBs speculatively offload to the gpu (with coherent memory (HSA, etc) access) at least under some constraints/patterns?
Re: Using SIMD to aggregate billions of values per second
#16I assume the values must be all be in memory beforehand and not hard storage.
Re: Using SIMD to aggregate billions of values per second
#17Earlier quoted context omitted.
There is no dedicated data compression at the moment as our use-case is centered around performance. That being said, we store data very efficiently with minimal overhead. One interesting consequence is that in many cases, we don't require indexes where other databases do. This synthetically compress the data relative to other DBs by removing the space taken by these indexes while improving query speed. It also means…
> we store data very efficiently with minimal overhead. Do you use encoding like delta-of-delta timestamps or something similar? > One interesting consequence is that in many cases, we don't require indexes where other databases do. I don't follow. Why don't you require indexes where other databases do? Thanks.
We don't store deltas, timestamps are stored as 64-bit int back to back in column. So are other primitives actually.
To avoid using indexes we store timestamps in ascending order. Timestamp interval search uses partitioning to cull chunks of data before lifting data in memory. Once in memory we use binary search on ordered timestamps to find intervals.
That said we do support indexes for key-based lookups
Re: Using SIMD to aggregate billions of values per second
#18In the reference there's no mention of SQL Window functions. Is it possible to do multiple moving averages over different time spans?
If not, are there plans to add support in the future?
Re: Using SIMD to aggregate billions of values per second
#19I assume the values must be all be in memory beforehand and not hard storage.
That sum() SQL is all you do as a user. There is no additional magic there.
The performance numbers are indeed best when data is in memory. However in reality sum() goes over memory mapped file, lazy loading data as required.
Re: Using SIMD to aggregate billions of values per second
#20In the reference there's no mention of SQL Window functions. Is it possible to do multiple moving averages over different time spans? If not, are there plans to add support in the future?
Window functions are in draft, we will release them imminently. We will support moving averages. In fact we plan to support generic multi-pass and window functions. Having multi-pass will allow you to do things like `select sum(x -sum(x)) from tab`.