not to hijack too much, but since this is on the topic of timeseries...i'm currently working on a fast* Canvas2D timeseries chart:
https://github.com/leeoniya/uPlot
* ~4,000 pts/ms on an i5 and integrated gpu
Using SIMD to aggregate billions of values per second
21–30 of 74 posts
Re: Using SIMD to aggregate billions of values per second
#22QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
This looks super interesting!
1) How does this compare to BlazingDB/MapD/etc, especially in queries on per $ per hour on AWS?
2) I don't see a way to give you money. What ensures that QuestDB will be around in the future if I architect my data pipelines around this product?
Re: Using SIMD to aggregate billions of values per second
#23QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
I'm curious what your business model is. Don't you worry that releasing your product for free will cut into your revenue?
Re: Using SIMD to aggregate billions of values per second
#24QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
This looks super interesting! 1) How does this compare to BlazingDB/MapD/etc, especially in queries on per $ per hour on AWS? 2) I don't see a way to give you money. What ensures that QuestDB will be around in the future if I architect my data pipelines around this product?
Thank you!
1) We didn't compare directly, but our aim is to squeeze as much performance out of hardware as possible to minimize cloud cost. If we are not the best at this right now - we are going to be the best.
2) We are VC funded with 24 months runway. The product is Open Source Apache 2.0 and will remain as such forever. Any future commercial offering will use open source product as a library.
Re: Using SIMD to aggregate billions of values per second
#25not to hijack too much, but since this is on the topic of timeseries...i'm currently working on a fast* Canvas2D timeseries chart: https://github.com/leeoniya/uPlot * ~4,000 pts/ms on an i5 and integrated gpu
This is super interesting and quite relevant to a bunch of stuff I do at work.
Next time I have a project that involves real-time plotting of data, I'll definitely take a look!
Re: Using SIMD to aggregate billions of values per second
#26What dialect of SQL are you using?
Re: Using SIMD to aggregate billions of values per second
#27What dialect of SQL are you using?
We try to stick to ANSI SQL as best as we can, but we have own own dialect. This should be a good entry point: https://www.questdb.io/docs/select
Re: Using SIMD to aggregate billions of values per second
#28I came across QuestDB in the past, but never tried myself. At my company, we use kx and onetick. Could you please elaborate why you are also comparing with Postgres since it's not really a time-series database nor revendicating to be part of the "high performance" club?
because they seem to only support that, from their page:
"As of now, SIMD operations are available for non-keyed aggregation queries, such as select sum(value) from table."
not even sure if they support where clauses on that, sums of functions of a column, or even other things like stddev of the column.
their storage format though looks good and simple (similar to kdb actually), but they really should have an 8-byte char instead of the 16-byte one since that would be far more used.
their partitioning scheme is only on time so less advanced than other system.
single designated timestamp column (so no bitemporal), but do support asof joins which is nice.
they totally screwed up on dates and time. dates only to milli and timestamps only to micros. huge mistake.
long256 which is nice, but strangely no long128s (which wind up being nice when you have base-10 fixed-point numbers normalized to a large number of decimals).
i didn't see any fixed-width string/byte columns. Does have 32-bit symbols (i assume similar to kdb?) that might cover some of those use cases.
some good and some bad in there. never going to compete with kdb or onetick on performance (and nobody competes with kdb on query language/extensibity) , but could find a niche based on price and having simpler more easily adapted to querying and more human interface.
Re: Using SIMD to aggregate billions of values per second
#29QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…
I'm curious what your business model is. Don't you worry that releasing your product for free will cut into your revenue?
In the future, we will release other products with features geared towards enterprises. We don't think that having our code open will harm the business.
On the contrary, it will hopefully help us gain some popularity by allowing any developer to access latency levels only seen in trading. We hope that QuestDB will allow them to stop worrying about database performance bottlenecks.
Re: Using SIMD to aggregate billions of values per second
#30Super interesting product I'll definitely be taking a deeper look at this when I'm at work tomorrow.
I notice all your comparisons are with floating point and integer types. I was recently looking at SIMD as a possible way to speed up some of our calculations. But we create financial software and most data is typically stored as decimals not floating points to avoid problems with binary floating point precision during calculations.
Does quest handle decimals, just without the SIMD speed up?
Is this just a dead end? Are there any SIMD implementations that deal with decimal numbers? I considered Hacky workarounds like using integer based types internally and then treating them as as fixed point decimals for display but that doesn't give enough range for my purposes.