Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#101

Post by bluestreak »

pachico wrote: post #94
bluestreak wrote: post #89

Earlier quoted context omitted.

Thank you for the kind words and constructive feedback. We are here to build on feedback like this. Grafana plugin is coming soon.

Glad to be useful. On the other side, I can tell you that ClickHouse also misses a feature everyone in the community of users wish for, which is automatic regarding when you add a new node (sort of what elasticsearch does). And before I forget, ClickHouse Kafka Engine is simply brilliant. The possibility of just publishing to Kafka and have your data not only inserted in your DB but also pre-processed is very powerfu…

This is incredibly useful, thank you! It would be awesome if we could chat more about your use cases at some point. Drop us a line on hello at questdb.io or join our slack. Whichever is easier for you.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#103

Post by ignoramous »

roskilli wrote: post #100
hintymad wrote: post #88

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless. A common flaw I see in many time-serie…

FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate multiple values/timeseries with a single set of dimensions if you use Protobuf's to write data, for more see the storage engine documentation[0]. The current recommendation is not to limit your queries but limit the global data queried per second[1] by a single…

Recommended reading on FST for the curious: https://blog.burntsushi.net/transducers/

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#104

Post by bluestreak »

hawk_ wrote: post #99
mpsq wrote: post #32

Earlier quoted context omitted.

As you said, performance is the main differentiator. We are orders of magnitude faster than TimescaleDB and InfluxDB on both data ingestion and querying. TimescaleDB relies on Postgres and has great SQL support. This is not the case for InfluxDB and this is where QuestDB shines: we do not plan to move away from SQL, we are very dedicated in bringing good support and some enhancements to make sure the querying languag…

do you do realtime steaming using SQL as well?

Over the network streaming is not yet available. Someone has mentioned Kafka support, how useful would that be to stream processed (aggregated) values and/or actual table changes?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#105

Post by santafen »

pachico wrote: post #94
bluestreak wrote: post #89

Earlier quoted context omitted.

Thank you for the kind words and constructive feedback. We are here to build on feedback like this. Grafana plugin is coming soon.

Glad to be useful. On the other side, I can tell you that ClickHouse also misses a feature everyone in the community of users wish for, which is automatic regarding when you add a new node (sort of what elasticsearch does). And before I forget, ClickHouse Kafka Engine is simply brilliant. The possibility of just publishing to Kafka and have your data not only inserted in your DB but also pre-processed is very powerfu…

Thanks for the helpful feedback! Feel free to reach out to chat more. I'm super interested in more feedback from you. davidgs(at)questdb(dot)io

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#107

Post by roskilli »

ignoramous wrote: post #103
roskilli wrote: post #100

Earlier quoted context omitted.

FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate multiple values/timeseries with a single set of dimensions if you use Protobuf's to write data, for more see the storage engine documentation[0]. The current recommendation is not to limit your queries but limit the global data queried per second[1] by a single…

Recommended reading on FST for the curious: https://blog.burntsushi.net/transducers/

Thank you for mentioning that, Andrew's post is really fantastic covering many things altogether: fundamentals, data structure, real world impact and examples.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#108

Post by roskilli »

bluestreak wrote: post #50
judofyr wrote: post #46

Congratulations on launching! It looks like a great product. Some technical questions which I didn’t see answered on my first glance: (1) Is it a single-server only, or is it possible to store data replicated as well? (2) I’m guessing that all the benchmarks were done with all the hot data paged into memory (correct?); what’s the performance once you hit the disk? How much memory do you recommend running with? (3) Ho…

thank you! - replication is in the works, this is going to be both TCP and UDP based, column-first, very fast. - yes, benchmarks are indeed are done on second pass over the mmaped pages. First pass would trigger IO, which is OS-driven and dependant on disk speed. We've seen well over 1.5Gb/s on disks that support this speed. Columns are mapped into memory separately and they are lazy accessed. So the memory footprint…

Curious: What is your strategy on replication? Is it some form of synchronous replication or asynchronous (i.e. active/passive with potential for data loss in event of hard loss of primary)? Also curious why you might look at UDP replication given unless using a protocol like QUIC on top of it, UDP replication would be inherently lossy (i.e. not even eventually consistent).

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#110

Post by bluestreak »

roskilli wrote: post #108
bluestreak wrote: post #50

Earlier quoted context omitted.

thank you! - replication is in the works, this is going to be both TCP and UDP based, column-first, very fast. - yes, benchmarks are indeed are done on second pass over the mmaped pages. First pass would trigger IO, which is OS-driven and dependant on disk speed. We've seen well over 1.5Gb/s on disks that support this speed. Columns are mapped into memory separately and they are lazy accessed. So the memory footprint…

Curious: What is your strategy on replication? Is it some form of synchronous replication or asynchronous (i.e. active/passive with potential for data loss in event of hard loss of primary)? Also curious why you might look at UDP replication given unless using a protocol like QUIC on top of it, UDP replication would be inherently lossy (i.e. not even eventually consistent).

The strategy is to multicast data to several nodes simultaneously. Data packets are sequence to allow receiver identify data loss. When loss is detected receiver finds breathing space to send a NACK. The packet and the nack would identify missing data chunk with O(1) complexity and sender then re-sends. Overall this method is lossless and avoids overhead of contacting nodes individually and sending same data over the network multiple times. This is useful in scenarios where several nodes participate in query execution and getting them up to date quickly is important.
Post reply on HN