Live data from Hacker News

Using SIMD to aggregate billions of values per second

questdb.io

41–50 of 74 posts

Re: Using SIMD to aggregate billions of values per second

#41

Earlier quoted context omitted.

Thanks! What sort of decimal type is it?

Well in code we're using C#'s decimal type (128 bit. 96 bits are used for an integer and the rest used for the sign and scaling factor) [0]. It's essentially just floating point applied to a base 10 integer rather than a binary one. In the SQL server database the column types are usually decimal(18,5) or decimal(25,12) [1] [0] - https://docs.microsoft.com/en-us/dotnet/api/system.decimal?v... [1] - https://docs.micros…

thanks! Someone else also pointed at lack of 128-bit long support. It looks as if long128 might help here.

Re: Using SIMD to aggregate billions of values per second

#43

Earlier quoted context omitted.

because they seem to only support that, from their page: "As of now, SIMD operations are available for non-keyed aggregation queries, such as select sum(value) from table." not even sure if they support where clauses on that, sums of functions of a column, or even other things like stddev of the column. their storage format though looks good and simple (similar to kdb actually), but they really should have an 8-byte…

Good summary, thank you. - we will extend SIMD to where clause, keyed aggregations, sampling, ordering, joins etc. It is a matter of time. - do you mind elaborating on how we screwed up date and time? - what makes you think we are never going to compete on performance with kdb+?

- nano are important for keeping ordering. while you may return results in insert order it is nice to have them so any other operations done in them outside the db you can retain (or recreate) that ordering. in financial systems nanos have become a sort of defacto standard for this. for examsple, all our messaging timestamps at places i've worked are always nano for anything written in the last 5-10 years.

Also when you are trying to do calculations on high-frequency data (tick, iot) it ruins your ability to take meaningful deltas (eg, arrival rates) since you get a lot of 0s and 1s for the time deltas. Its difficult to take weighted averages with weights of 0s.

the issues solving that (if you really need a wider range) are easier to solve that having to force everything down to micros and creating ways around that. (eg, kdb uses multiple date, time, and timestamp types and it doesn't use the unix epoch since it isn't very useful for tick, censor, or any high-frequency data i've seen).

better than a double that some systems still use.

-kdb's secret sauce that people don't seem to understand is its query language that more naturally fits the time series domain. It isn't really a database as more it is an array language with a database component. (eg, try to write an efficient sql query that calculates 5 minute bars on tick data).

I actually like Java too - I've written or worked on a couple trading systems written in core java. just get good devs who understand what it means to write zero gc code, abuse off-heap memory, and understand what hotspot can intrinsify. If you can stay in the simd code for all the heavy lifting loops (filters, aggregates, etc), I don't think java will be an impediment.

I think you have parts going in the correct direction, and you seem to have good experience from looking at the bios. Nothing really un-fixable (or un-addable) in what I saw glancing at your docs. I did bookmark you to see how the db goes. Will prob check out soon.

Re: Using SIMD to aggregate billions of values per second

#44

Earlier quoted context omitted.

because they seem to only support that, from their page: "As of now, SIMD operations are available for non-keyed aggregation queries, such as select sum(value) from table." not even sure if they support where clauses on that, sums of functions of a column, or even other things like stddev of the column. their storage format though looks good and simple (similar to kdb actually), but they really should have an 8-byte…

Good summary, thank you. - we will extend SIMD to where clause, keyed aggregations, sampling, ordering, joins etc. It is a matter of time. - do you mind elaborating on how we screwed up date and time? - what makes you think we are never going to compete on performance with kdb+?

The reason why you cannot ever compete with kdb+/q is because the database and language run in one address space. Your benchmark gets around this problem by using the built in sum() function, but kdb+/q can just execute arbitrary code and never suffer a performance penalty. Unless you plan on integrated a high performance programming language into your DB, it simply will not be possible to ever meaningfully compete on the effective total time of complex queries.

I, of course, am not disparaging your work, the performance numbers are very impressive!

Re: Using SIMD to aggregate billions of values per second

#45
This can't be called a database due to lacking of any persistent storage. It does not survive a system crash.

It is a structured in-memory (or rather in-JVM) cache with a rudimentary SQL interface.

Calling things by its proper names is a half-way to intelligence.

Re: Using SIMD to aggregate billions of values per second

#47

not to hijack too much, but since this is on the topic of timeseries...i'm currently working on a fast* Canvas2D timeseries chart: https://github.com/leeoniya/uPlot * ~4,000 pts/ms on an i5 and integrated gpu

this is pretty cool!

Re: Using SIMD to aggregate billions of values per second

#49
post #20

In the reference there's no mention of SQL Window functions. Is it possible to do multiple moving averages over different time spans? If not, are there plans to add support in the future?

Window functions are in draft, we will release them imminently. We will support moving averages. In fact we plan to support generic multi-pass and window functions. Having multi-pass will allow you to do things like `select sum(x -sum(x)) from tab`.

those things are so useless if your time points arent equidistant (ie, you don't care about the last 100 rows, you care about the last 5 minutes). they basically force you to use very slow correlated subqueries. please do something better.

Re: Using SIMD to aggregate billions of values per second

#50

QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…

Just wanted to point out a small typo in the docs: "LASTEST BY" (https://www.questdb.io/docs/crudOperations)
Post reply on HN