Using SIMD to aggregate billions of values per second • Hacker News Board

Re: Using SIMD to aggregate billions of values per second

#71

Post by bluestreak » Wed, Apr 08, 2020, 2:13 PM UTC

binomiq wrote: post #69

bluestreak wrote: post #57

Earlier quoted context omitted.

these are against the latest 4.0 KDB+, after 26 March. KDB before that could not aggregate in parallel implicitly.

In your benchmarks, KDB's max on longs takes twice as long as sum on longs. I am not able to replicate this with the most recent version of KDB (2020.03.30). $QHOME/l64/q -s 4 KDB+ 4.0 2020.03.30 Copyright (C) 1993-2020 Kx Systems l64/ 4()core 516718MB q)zz:1000000000?1000j q)0.01*system"t do[100;max zz]" 257.52 q)0.01*system"t do[100;sum zz]" 251.95 There's a marginal difference here rather than double. For the vers…

sorry, i think we tested with this one:

    KDB+ 4.0 2020.03.17 Copyright (C) 1993-2020 Kx Systems
    l64/ 12(16)core 63960MB
    EXPIRE 2021.03.26

Re: Using SIMD to aggregate billions of values per second

#72

Post by dintech » Wed, Apr 08, 2020, 2:20 PM UTC

bluestreak wrote: post #71

binomiq wrote: post #69

Earlier quoted context omitted.

In your benchmarks, KDB's max on longs takes twice as long as sum on longs. I am not able to replicate this with the most recent version of KDB (2020.03.30). $QHOME/l64/q -s 4 KDB+ 4.0 2020.03.30 Copyright (C) 1993-2020 Kx Systems l64/ 4()core 516718MB q)zz:1000000000?1000j q)0.01*system"t do[100;max zz]" 257.52 q)0.01*system"t do[100;sum zz]" 251.95 There's a marginal difference here rather than double. For the vers…

sorry, i think we tested with this one: KDB+ 4.0 2020.03.17 Copyright (C) 1993-2020 Kx Systems l64/ 12(16)core 63960MB EXPIRE 2021.03.26

Even without that 2x, all benchmarks are still in the same ballpark which is impressive in its own right. It will be interesting to see where this goes.

Re: Using SIMD to aggregate billions of values per second

#73

Post by smabie » Wed, Apr 08, 2020, 6:52 PM UTC

patrick73_uk wrote: post #54

smabie wrote: post #44

Earlier quoted context omitted.

The reason why you cannot ever compete with kdb+/q is because the database and language run in one address space. Your benchmark gets around this problem by using the built in sum() function, but kdb+/q can just execute arbitrary code and never suffer a performance penalty. Unless you plan on integrated a high performance programming language into your DB, it simply will not be possible to ever meaningfully compete o…

I'm a QuestDB dev, data on a QuestDb is also stored in a single address space and SQL queries are compiled into objects that run in that address space. However, it is just SQL not a bespoke language. A future possibility would be to allow queries in java or scala.

Implementing the ability for QuestDB to dynamically load jars would be really cool. And if you exposed an interface to directly communicate with the Db, you could get rid of the SQL parsing overhead as well. This would also allow QuestDB to function as an app engine of sorts, just like kdb+/q. I see real value in that for latency sensitive financial applications.

Re: Using SIMD to aggregate billions of values per second

#74

Post by jnordwick » Thu, Apr 09, 2020, 7:01 AM UTC

bluestreak wrote: post #68

salmonlogs wrote: post #64

Earlier quoted context omitted.

"its not possible to be accurate to that resolution with current hardware" Are you referring to the clock precision of consumer grade hardware here? In my experience the vast majority of financial time series data is reported in nanoseconds. The data providers, vendors, exchanges and data brokers absolutely have hardware capable of measuring timestamps in nanoseconds. The accuracy doesn't have to be to 1ns of resolut…

We are going to add a new type in the future to support nanos! Sorry for the confusion.

can you please contact me to jnordwick@gmail.com?