Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#121

Post by j1897 »

gregmac wrote: post #120
shay_ker wrote: post #12

Absolutely love the story. TimescaleDB & InfluxDB have had a lot of posts on HN, so I'm sure others are wondering - how do we compare QuestDB to them? It sounds like performance is a big one, but I'm curious to hear your take on it.

Is also be interested in hearing when is QuestDB not a good choice? Are there use cases where TimescaleDB, InfluxDB, ClickHouse or something else are better suited?

Hard question to answer because each solution is unique and has its own tradeoffs. Taking a step back QuestDB is a less mature product than the ones mentioned, and therefore there are many features, integrations etc. to build on our side. This is a reflection of how long we have been around and capital we have raised versus those companies who are much larger in size.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#122

Post by bluestreak »

monstrado wrote: post #115
bluestreak wrote: post #76

Earlier quoted context omitted.

There will be two different flavors of replication: - TCP-based replication for WAN - UDP-based replication for LAN and high traffic environments We are currently building foundation elements of this replication, such as column-first and parallel writes. These will go into and always be part of QuestDB. TCP-replication will go on top of this foundation and also part of QuestDB. UDP-based replication will be a part of…

Thanks for your response! Last question... Will the clustering target just replication (HA) or will it also target sharding for scaling out storage capacity?

:)

Eventually both. We are starting with baby steps, e.g. get data from A to B quickly and reliably. Replication/HA will be first of course. Then we want to scale queries across multiple hosts. Since all nodes have the same data - they may as well all participate. Sharding will be last. We are thinking of taking a route of virtualizing tables. Each shard can be its own table and SQL optimiser can use them as partitions of single virtual table. We already take single table and partition it for execution. Sharding seems almost like a natural fit.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#123

Post by bluestreak »

gregmac wrote: post #120
shay_ker wrote: post #12

Absolutely love the story. TimescaleDB & InfluxDB have had a lot of posts on HN, so I'm sure others are wondering - how do we compare QuestDB to them? It sounds like performance is a big one, but I'm curious to hear your take on it.

Is also be interested in hearing when is QuestDB not a good choice? Are there use cases where TimescaleDB, InfluxDB, ClickHouse or something else are better suited?

OLTP is not a good fit, if your workflow consists from INSERT/UPDATE/DELETE statements

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#124

Post by santafen »

TheRealNGenius wrote: post #119

Maybe I'm out of the loop, but I noticed lately that a majority of show/launch hn posts I click on have text that is muted. I know this happens on down voted comments, but is this saying that people are down voting the post itself?

I don’t think it’s being downvoted. Maybe it’s because it’s an actual post?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#125

Post by bluestreak »

vii wrote: post #118
bluestreak wrote: post #42

Earlier quoted context omitted.

thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places where we could not beat aggregation of julia, but the delta wasn't very big. This could have been down to mapped memory. We will definitely try things with direct memory too!

The databases I implemented experimented with various ways to compile queries. Turns out that the JVM can run quite fast. I feel like LLVM (Julia) is likely to be able to be better for throughput and definitely better for predictability of performance. If you know layouts and sizes, then your generated code can run really fast - using SIMD and not checking bounds is a win. Hugepages would greatly reduce pagetable boo…

Our best performance currently is in C++ code and LLVM is something we are considering using to compute expressions, such as predicates and select clause items. This is most likely to be way faster than what we can currently do in Java. What I would like to know if LLVM can optimize all the way to AVX512?

We also need to experiment with hugepages. The beauty is that if read and write are separated - there is no issue with writes. They can still use 4k pages!

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#127

Post by bluestreak »

architectonic wrote: post #116
bluestreak wrote: post #93

Earlier quoted context omitted.

We store "dimensions" as table columns with no artificial limits on column count. If you able to send all dimensions in the same message, they will be stored on one row of data. If dimensions are sent as separate messages, current implementation will store them on different rows. This will make columns sparse. We can change that if need be and "update" the same row as dimensions arrive as long as they have the same t…

Can you handle multiple time dimensions efficiently? We have 3 of them, can one get away without having to physically store "slices" on one of them?

if you can send all three in the same message, Influx Line Protocol for example, we will store them as 3 columns in one table. Does this help?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#129

Post by beagle3 »

thegreatpeter wrote: post #13

Am I the only one that's like "wtf is a time-series database compared to a normal one?"

One differentiating feature is "as of" join. You have records of the form (time, value), and you ask "what's the most recent value as of $time?"; On a non-TS oriented DBMS, this query is usually slow and hard to write. Window extensions to SQL can make it a little better, but - you can assume that a proper TSDB answers this query x10 to x10,000 times faster on the same hardware, especially when done in bulk (e.g.: I have one million (time,bid_price) records, and one million (time,transaction_price) records; For each transaction record, I want to know what the most-recent bid price was at that time.

That's something kdb+ and ClickHouse do in milliseconds; and I assume QuestDB can to, though I didn't check.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#130

Post by mfreed »

mpsq wrote: post #32
shay_ker wrote: post #12

Absolutely love the story. TimescaleDB & InfluxDB have had a lot of posts on HN, so I'm sure others are wondering - how do we compare QuestDB to them? It sounds like performance is a big one, but I'm curious to hear your take on it.

As you said, performance is the main differentiator. We are orders of magnitude faster than TimescaleDB and InfluxDB on both data ingestion and querying. TimescaleDB relies on Postgres and has great SQL support. This is not the case for InfluxDB and this is where QuestDB shines: we do not plan to move away from SQL, we are very dedicated in bringing good support and some enhancements to make sure the querying languag…

Hi, TimescaleDB cofounder here. Nice to read about your journey in time-series data, and always welcome another database that can satisfy a specific type of developer needs.

I also commend you on your desire to rebuild everything Postgres offers from scratch. We took a different route by building on top of Postgres (which e.g. allowed us to launch with native replication, rock-solid reliability, window functions, geo spatial data, etc without sacrificing performance). But there are many ways up this mountain!

As a quick thing, however: While it’s not very representative of the workloads we typically see, I tried your simple 1B scan on a minimally-configured hypertable in TimescaleDB/PostgreSQL, and got results that were >12x faster on my 8-core laptop than what you were reporting on a 48-core AWS m5.metal instance.

I think the Hacker News community always appreciates transparency in benchmarking; looking forward to reading a follow up post where you share reproducible benchmarks where all databases are tuned equivalently.

Post reply on HN