Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#111

Post by santafen »

bravura wrote: post #109

Can you talk about some of the ideal use cases for a time series db? Versus Postgres or a graph database.

Great question! Time series databases are a great solution for applications that need to process streams of data. IoT is a popular use case. DevOps and infrastructure monitoring applications as well. As has been mentioned in other comments here, there are a lot of use cases in financial services as well.

These are all applications where you’re dealing with streams of time-stamped data that needs to be ingested, stored, and queried in huge volumes.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#112

Post by hintymad »

roskilli wrote: post #100
hintymad wrote: post #88

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless. A common flaw I see in many time-serie…

FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate multiple values/timeseries with a single set of dimensions if you use Protobuf's to write data, for more see the storage engine documentation[0]. The current recommendation is not to limit your queries but limit the global data queried per second[1] by a single…

Thanks, @roskilli! Nice documentation.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#113

Post by pachico »

santafen wrote: post #105
pachico wrote: post #94

Earlier quoted context omitted.

Glad to be useful. On the other side, I can tell you that ClickHouse also misses a feature everyone in the community of users wish for, which is automatic regarding when you add a new node (sort of what elasticsearch does). And before I forget, ClickHouse Kafka Engine is simply brilliant. The possibility of just publishing to Kafka and have your data not only inserted in your DB but also pre-processed is very powerfu…

Thanks for the helpful feedback! Feel free to reach out to chat more. I'm super interested in more feedback from you. davidgs(at)questdb(dot)io

I certainly will. Cheers

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#114

Post by hintymad »

bluestreak wrote: post #93
hintymad wrote: post #88

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless. A common flaw I see in many time-serie…

We store "dimensions" as table columns with no artificial limits on column count. If you able to send all dimensions in the same message, they will be stored on one row of data. If dimensions are sent as separate messages, current implementation will store them on different rows. This will make columns sparse. We can change that if need be and "update" the same row as dimensions arrive as long as they have the same t…

Thanks for the explanation.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#115

Post by monstrado »

bluestreak wrote: post #76
monstrado wrote: post #70

I noticed there is "Clustering" mentioned under enterprise features, but I can't seem to find any references to it in the documentation. Is this something that will be strictly closed source?

There will be two different flavors of replication: - TCP-based replication for WAN - UDP-based replication for LAN and high traffic environments We are currently building foundation elements of this replication, such as column-first and parallel writes. These will go into and always be part of QuestDB. TCP-replication will go on top of this foundation and also part of QuestDB. UDP-based replication will be a part of…

Thanks for your response! Last question...

Will the clustering target just replication (HA) or will it also target sharding for scaling out storage capacity?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#116

Post by architectonic »

bluestreak wrote: post #93
hintymad wrote: post #88

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless. A common flaw I see in many time-serie…

We store "dimensions" as table columns with no artificial limits on column count. If you able to send all dimensions in the same message, they will be stored on one row of data. If dimensions are sent as separate messages, current implementation will store them on different rows. This will make columns sparse. We can change that if need be and "update" the same row as dimensions arrive as long as they have the same t…

Can you handle multiple time dimensions efficiently? We have 3 of them, can one get away without having to physically store "slices" on one of them?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#117

Post by santafen »

120bits wrote: post #95
Random_ernest wrote: post #30

Testing out the demo: SELECT * FROM trips WHERE tip_amount > 500 ORDER BY tip_amount DESC Very interesting :-)

For some reason this query is taking too long to execute. Not sure if I missed something.

When I ran it it took about 20s total.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#118

Post by vii »

bluestreak wrote: post #42
vii wrote: post #28

mmap'd databases are really quick to implement. I implemented both row and column orientated databases. The traders and quants loved it - and adoption took off after we built a web interface that let you see a whole day and also zoom into exact trades with 100ms load times for even the most heavily traded symbols. The benefits of mmaping and in general POSIX filesystem atomic properties are quick implementation, wher…

thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places where we could not beat aggregation of julia, but the delta wasn't very big. This could have been down to mapped memory. We will definitely try things with direct memory too!

The databases I implemented experimented with various ways to compile queries. Turns out that the JVM can run quite fast. I feel like LLVM (Julia) is likely to be able to be better for throughput and definitely better for predictability of performance.

If you know layouts and sizes, then your generated code can run really fast - using SIMD and not checking bounds is a win.

Hugepages would greatly reduce pagetable bookkeeping, but obviously may magnify writes. Wish I could have tried that!

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#120

Post by gregmac »

shay_ker wrote: post #12

Absolutely love the story. TimescaleDB & InfluxDB have had a lot of posts on HN, so I'm sure others are wondering - how do we compare QuestDB to them? It sounds like performance is a big one, but I'm curious to hear your take on it.

Is also be interested in hearing when is QuestDB not a good choice? Are there use cases where TimescaleDB, InfluxDB, ClickHouse or something else are better suited?
Post reply on HN