Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using SIMD to aggregate billions of values per second

questdb.io

Post reply on HN
Print view View original

61–70 of 74 posts

Re: Using SIMD to aggregate billions of values per second

#61

Post by throwaway_pdp09 »

SimonPStevens wrote: post #30

Super interesting product I'll definitely be taking a deeper look at this when I'm at work tomorrow. I notice all your comparisons are with floating point and integer types. I was recently looking at SIMD as a possible way to speed up some of our calculations. But we create financial software and most data is typically stored as decimals not floating points to avoid problems with binary floating point precision durin…

Can't you just use a binary int with the few bits being an implicit fraction (or just count in 'pennies' AKA the lowest denominction of that currency). Seems a straightforward solution, no?

Re: Using SIMD to aggregate billions of values per second

#62

Post by dkersten »

bluestreak wrote: post #24
lsb wrote: post #22

Earlier quoted context omitted.

This looks super interesting! 1) How does this compare to BlazingDB/MapD/etc, especially in queries on per $ per hour on AWS? 2) I don't see a way to give you money. What ensures that QuestDB will be around in the future if I architect my data pipelines around this product?

Thank you! 1) We didn't compare directly, but our aim is to squeeze as much performance out of hardware as possible to minimize cloud cost. If we are not the best at this right now - we are going to be the best. 2) We are VC funded with 24 months runway. The product is Open Source Apache 2.0 and will remain as such forever. Any future commercial offering will use open source product as a library.

On point 2, I hope you have a good monetisation strategy. I’m a bit wary of new open source databases that don’t have a solid strategy for when their funding runs out, too many times have projects been more or less abandoned after funding ran out, despite being open source (eg Rethinkdb), so I’m rather wary of relying on something until it’s been well established. (Although even basho eventually shut down despite being well established...)

Re: Using SIMD to aggregate billions of values per second

#63

Post by bluestreak »

numlock86 wrote: post #55
bluestreak wrote: post #2

QuestDB co-founder and CTO here - happy to share questdb, a performance-driven open-source time-series database that uses SQL. High performance databases have a reputation of being inaccessible. They are expensive, closed-source, and require complex proprietary languages. We have made our code available under Apache 2.0. Under this new release, QuestDB leverages SIMD instructions, vectorizations and parallel executio…

Serious but bold question: What are the benefits versus Clickhouse for example? Why should I use QuestDB? Amazing work either way. The space of databases can never have too much competition.

It is hard for me to say right now as we did not benchmark against Clickhouse yet, this is clearly the most requested comparison. We will come back on this!

Re: Using SIMD to aggregate billions of values per second

#64

Post by salmonlogs »

patrick73_uk wrote: post #53
jnordwick wrote: post #43

Earlier quoted context omitted.

- nano are important for keeping ordering. while you may return results in insert order it is nice to have them so any other operations done in them outside the db you can retain (or recreate) that ordering. in financial systems nanos have become a sort of defacto standard for this. for examsple, all our messaging timestamps at places i've worked are always nano for anything written in the last 5-10 years. Also when…

I'm a QuestDB dev, with regards to nano timestamps, we dont use a nanosecond timestamp because its not possible to be accurate to that resolution with current hardware. However, on a single host the nano second clocks are precise and monotonic, they would be useful to maintain order. I think they do make sense and we will have to look into providing timestamps to that resolution.

"its not possible to be accurate to that resolution with current hardware"

Are you referring to the clock precision of consumer grade hardware here?

In my experience the vast majority of financial time series data is reported in nanoseconds. The data providers, vendors, exchanges and data brokers absolutely have hardware capable of measuring timestamps in nanoseconds.

The accuracy doesn't have to be to 1ns of resolution to warrant measuring in nanos - even to the nearest 100ns is a useful and meaningful improvement beyond micros.

Re: Using SIMD to aggregate billions of values per second

#65

Post by bufferoverflow »

leeoniya wrote: post #21

not to hijack too much, but since this is on the topic of timeseries...i'm currently working on a fast* Canvas2D timeseries chart: https://github.com/leeoniya/uPlot * ~4,000 pts/ms on an i5 and integrated gpu

Why canvas2d when WebGL is much faster?

Re: Using SIMD to aggregate billions of values per second

#66

Post by binomiq »

patrick73_uk wrote: post #53
jnordwick wrote: post #43

Earlier quoted context omitted.

- nano are important for keeping ordering. while you may return results in insert order it is nice to have them so any other operations done in them outside the db you can retain (or recreate) that ordering. in financial systems nanos have become a sort of defacto standard for this. for examsple, all our messaging timestamps at places i've worked are always nano for anything written in the last 5-10 years. Also when…

I'm a QuestDB dev, with regards to nano timestamps, we dont use a nanosecond timestamp because its not possible to be accurate to that resolution with current hardware. However, on a single host the nano second clocks are precise and monotonic, they would be useful to maintain order. I think they do make sense and we will have to look into providing timestamps to that resolution.

KDB works around this by storing the nanos in 64 bits but only for a particular time range.

1707.09.22D00:12:43.145224194 (max negative) to 2292.04.10D23:47:16.854775806 (max positive)

With 0 at 2000.01.01D00:00:00.000000000

Re: Using SIMD to aggregate billions of values per second

#67

Post by [deleted] »

dkersten wrote: post #62
bluestreak wrote: post #24

Earlier quoted context omitted.

Thank you! 1) We didn't compare directly, but our aim is to squeeze as much performance out of hardware as possible to minimize cloud cost. If we are not the best at this right now - we are going to be the best. 2) We are VC funded with 24 months runway. The product is Open Source Apache 2.0 and will remain as such forever. Any future commercial offering will use open source product as a library.

On point 2, I hope you have a good monetisation strategy. I’m a bit wary of new open source databases that don’t have a solid strategy for when their funding runs out, too many times have projects been more or less abandoned after funding ran out, despite being open source (eg Rethinkdb), so I’m rather wary of relying on something until it’s been well established. (Although even basho eventually shut down despite bei…

[deleted]

Re: Using SIMD to aggregate billions of values per second

#68

Post by bluestreak »

salmonlogs wrote: post #64
patrick73_uk wrote: post #53

Earlier quoted context omitted.

I'm a QuestDB dev, with regards to nano timestamps, we dont use a nanosecond timestamp because its not possible to be accurate to that resolution with current hardware. However, on a single host the nano second clocks are precise and monotonic, they would be useful to maintain order. I think they do make sense and we will have to look into providing timestamps to that resolution.

"its not possible to be accurate to that resolution with current hardware" Are you referring to the clock precision of consumer grade hardware here? In my experience the vast majority of financial time series data is reported in nanoseconds. The data providers, vendors, exchanges and data brokers absolutely have hardware capable of measuring timestamps in nanoseconds. The accuracy doesn't have to be to 1ns of resolut…

We are going to add a new type in the future to support nanos! Sorry for the confusion.

Re: Using SIMD to aggregate billions of values per second

#69

Post by binomiq »

bluestreak wrote: post #57
binomiq wrote: post #56

Were these benchmarks before or after 2020.03.26? There was a bug that caused max operations to take twice as long. From the KDB+ 4.0 release notes: 2020.03.26 FIX fixed performance regression for max. e.g. q)x:100000000?100;system"ts:10 max x"

these are against the latest 4.0 KDB+, after 26 March. KDB before that could not aggregate in parallel implicitly.

In your benchmarks, KDB's max on longs takes twice as long as sum on longs. I am not able to replicate this with the most recent version of KDB (2020.03.30).

    $QHOME/l64/q -s 4
    KDB+ 4.0 2020.03.30 Copyright (C) 1993-2020 Kx Systems
    l64/ 4()core 516718MB 

    q)zz:1000000000?1000j
    q)0.01*system"t do[100;max zz]"
    257.52
    q)0.01*system"t do[100;sum zz]"
    251.95
There's a marginal difference here rather than double.

For the version of KDB with the regression bug:

    q)0.01*system"t do[100;max zz]"
    512.21
    q)0.01*system"t do[100;sum zz]"
    254.24
Which starts to look more like your numbers.

Re: Using SIMD to aggregate billions of values per second

#70

Post by leeoniya »

bufferoverflow wrote: post #65
leeoniya wrote: post #21

not to hijack too much, but since this is on the topic of timeseries...i'm currently working on a fast* Canvas2D timeseries chart: https://github.com/leeoniya/uPlot * ~4,000 pts/ms on an i5 and integrated gpu

Why canvas2d when WebGL is much faster?

a few reasons.

the main one is that i know canvas and can implement everything i need, but would need to learn a lot of webgl to get any further than a basic PoC.

browsers limit how many webgl contexts you can acquire. chrome is capped at 16, so if you need more than 16 charts on a page, you're out of luck.

initializing webgl is actually slower than canvas 2d.

it's difficult to draw lines of different thicknesses in raw webgl - it takes a lot of code.

there are projects which abstract webgl nicely to provide canvas-like ergonomics, like Two.js, Pixi.js, or Canvas2DtoWebGL/litegl [1], but they would make the codebase much larger and more complex.

[1] https://github.com/jagenjo/Canvas2DtoWebGL

Post reply on HN