Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#141

Post by j1897 »

mekster wrote: post #140
mpsq wrote: post #32

Earlier quoted context omitted.

As you said, performance is the main differentiator. We are orders of magnitude faster than TimescaleDB and InfluxDB on both data ingestion and querying. TimescaleDB relies on Postgres and has great SQL support. This is not the case for InfluxDB and this is where QuestDB shines: we do not plan to move away from SQL, we are very dedicated in bringing good support and some enhancements to make sure the querying languag…

Did you also compare to VictoriaMetrics?

Not yet - there is a bench vs clickhouse that has been done by one of their contributor though see below in the comments.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#143

Post by polskibus »

bluestreak wrote: post #42
vii wrote: post #28

mmap'd databases are really quick to implement. I implemented both row and column orientated databases. The traders and quants loved it - and adoption took off after we built a web interface that let you see a whole day and also zoom into exact trades with 100ms load times for even the most heavily traded symbols. The benefits of mmaping and in general POSIX filesystem atomic properties are quick implementation, wher…

thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places where we could not beat aggregation of julia, but the delta wasn't very big. This could have been down to mapped memory. We will definitely try things with direct memory too!

Please consider going through Andy Pavlo's superb video courses on building databases.

http://www.cs.cmu.edu/~pavlo/

He discusses many common pitfalls and architectural decisions that impact database quality and performance, including map and why it's bad.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#144

Post by tuananh »

120bits wrote: post #95
Random_ernest wrote: post #30

Testing out the demo: SELECT * FROM trips WHERE tip_amount > 500 ORDER BY tip_amount DESC Very interesting :-)

For some reason this query is taking too long to execute. Not sure if I missed something.

took almost a min for me

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#145

Post by sirffuzzylogik »

jankotek wrote: post #26

Good luck. I work on similar OS database engine for about decade now. It is not bad, but I think consulting is better way to get funds. Also avoid "zero gc", JVM can be surprisingly good. Will be in touch :)

Thanks Jan!

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#146

Post by sirffuzzylogik »

gregwebs wrote: post #51

I am still hoping to see comparisons to Victoria Metrics, which also shows much better performance than many other TSDB. Victoria Metrics is Prometheus compatible whereas Quest now supports Postgres compatibility. Both have compatibility with InfluxDB. The Victoria Metrics story is somewhat similar where someone tried using Clickhouse for large time series data at work and was astonished at how much faster it was. He…

This is on the roadmap, we will work on integrating with https://github.com/timescale/tsbs, TSBS has Victoria Metrics support too.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#147

Post by judofyr »

bluestreak wrote: post #110
roskilli wrote: post #108

Earlier quoted context omitted.

Curious: What is your strategy on replication? Is it some form of synchronous replication or asynchronous (i.e. active/passive with potential for data loss in event of hard loss of primary)? Also curious why you might look at UDP replication given unless using a protocol like QUIC on top of it, UDP replication would be inherently lossy (i.e. not even eventually consistent).

The strategy is to multicast data to several nodes simultaneously. Data packets are sequence to allow receiver identify data loss. When loss is detected receiver finds breathing space to send a NACK. The packet and the nack would identify missing data chunk with O(1) complexity and sender then re-sends. Overall this method is lossless and avoids overhead of contacting nodes individually and sending same data over the…

This reminds me a bit of Aeron (https://github.com/real-logic/aeron) which is a reliable UDP uni/multicast transport library with built-in flow control. It's written in Java and seems to have superb performance (I haven't used it myself). Might be an interesting alternative if you don't want to write it all yourself.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#148

Post by sirffuzzylogik »

srini20 wrote: post #96
sirffuzzylogik wrote: post #49

Earlier quoted context omitted.

j1897 is referring to https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-b...

(Hard to draw many meaningful conclusions from a single, extremely simple query without much explanation?) Graph shows PostgreSQL as taking a long time, but doesn't say anything about configuration or parallelization. PostgreSQL should be able to parallelize that type of query since 9.6+, but I think they didn't use parallelization in these experiments with PostgreSQL, even though they used a bunch of parallel thread…

I agree that our blog post lacks of details, here are some:

- PostgreSQL 12

- 12

- No

- We ran the test using the pg_prewarm [0] module, the difference was negligible

Regarding the "toy" query, the reason we are showcasing this instead of other more complex queries is because this is a simple, easily reproducible benchmark. It provides a point of reference for performance figures.

> Database benchmarking is pretty complex (and easy to bias), and most queries do not look like this toy one.

I would say that benchmarking is very hard. We tried not to perform a biased benchmark by running something that is not time-series specific and which does not put us in advantage compared to what Postgres should do.

The takeaway from this is that configuration is important and we should expose it. The next benchmark we do will have an associated repository so people can review our config and point non optimal items if any.

[0]: https://www.postgresql.org/docs/9.4/pgprewarm.html

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#149

Post by einpoklum »

beagle3 wrote: post #129
thegreatpeter wrote: post #13

Am I the only one that's like "wtf is a time-series database compared to a normal one?"

One differentiating feature is "as of" join. You have records of the form (time, value), and you ask "what's the most recent value as of $time?"; On a non-TS oriented DBMS, this query is usually slow and hard to write. Window extensions to SQL can make it a little better, but - you can assume that a proper TSDB answers this query x10 to x10,000 times faster on the same hardware, especially when done in bulk (e.g.: I…

> On a non-TS oriented DBMS, this query is usually slow and hard to write.

I don't see why this is hard to write:

SELECT max(time) as most_recent_time, transaction_price INTO newtable FROM oldtable WHERE time > $my_datetime GROUP BY time;

As for being slow - can you provide some references about this being slow in non-time-series columnar DBMSes? MonetDB, Vectorwise, Vertica?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#150

Post by einpoklum »

avthar wrote: post #31
thegreatpeter wrote: post #13

Am I the only one that's like "wtf is a time-series database compared to a normal one?"

This is actually an underrated question. Time-series databases offer better performance and usability for dealing with time-series data (think DevOps metrics, data from IoT devices, stock prices etc, anything where you're monitoring and analyzing how things change over time) They allow you answer questions where time is the main component of interest much more quickly and easily: eg 1: IoT Sensors) Show me the averag…

If you're comparing performance against a transaction-oriented DBMS like postgres, rather than an analytics-oriented columnar DBMS like Actian Vector, MonetDB, HP Vertica etc - then of course you'll get bad perfromance. The former kind are typically up to 1000x slower than the latter kind on analytic queries.
Post reply on HN