Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#91

Post by airstrike »

bluestreak wrote: post #87
airstrike wrote: post #86

There's an opportunity for a tool that combines this sort of technology in the backend with a spreadsheet-like GUI powered by formulas and all the user friendliness that comes with a non-programmer interface. Wall Street would forever be changed. Source: I'm one of the poor souls fighting my CPU and RAM to do the same thing with Excel and non-native add-ins by {FactSet, Capital IQ, Bloomberg} This stuff SELECT * FROM…

We are working on building a solid PostgreSQL support insofar as allowing ODBC driver to execute this type of query from Excel. This is work in progress with not that much left on it.

Awesome! I think about this almost on a daily basis, and could very well be wrong, but from my perspective think the killer feature is integrating the querying with the financial data providers I mentioned above so they could sell the whole thing as the final product to end users. (EDIT: from a reply to another comment, it seems like some people are onto the concept: https://factset.quantopian.com)

If you ever install FactSet for a trial period and try querying time series with even ~10,000+ data points, you'd be amazed at how long it takes, how sluggish it is and how often Excel crashes.

My real perspective is Microsoft should roll something similar out as part of Excel and also get in the business of providing the financial data as they continue the transition into services over products

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#92

Post by avthar »

airstrike wrote: post #80
avthar wrote: post #31

Earlier quoted context omitted.

This is actually an underrated question. Time-series databases offer better performance and usability for dealing with time-series data (think DevOps metrics, data from IoT devices, stock prices etc, anything where you're monitoring and analyzing how things change over time) They allow you answer questions where time is the main component of interest much more quickly and easily: eg 1: IoT Sensors) Show me the averag…

> eg 2: Financial data) What's the price of stock X over the past 5 years This is so incredibly frustratingly slow to pull on FactSet and Capital IQ, it makes me want to pull my hair every time I have to build line charts over time for a period greater than 2 years

Sounds like you need a time-seres database for those sorts of narrow and deep queries :)

What's difficult is to find a database that has good performance on both narrow and deep queries (e.g Price of stock X for past 5 years) as well as shallow and wide queries (e.g Price of all stocks in past 15mins)

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#93

Post by bluestreak »

hintymad wrote: post #88

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless. A common flaw I see in many time-serie…

We store "dimensions" as table columns with no artificial limits on column count. If you able to send all dimensions in the same message, they will be stored on one row of data. If dimensions are sent as separate messages, current implementation will store them on different rows. This will make columns sparse. We can change that if need be and "update" the same row as dimensions arrive as long as they have the same timestamp value.

There is an option to store set of dimensions separately as asof/splice join separate tables.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#94

Post by pachico »

bluestreak wrote: post #89
pachico wrote: post #84

I see this as a very interesting project. I use ClickHouse as OLAP and I'm very happy with it. I can tell you features that make me stick to it. If some day QuestDB offers them, I might explore the possibility to switch but never before. - very fast (I guess we're aligned here) - real time materialized views for aggregation functions (this is absolutely a killer feature that makes it quite pointless to be fast if you…

Thank you for the kind words and constructive feedback. We are here to build on feedback like this. Grafana plugin is coming soon.

Glad to be useful. On the other side, I can tell you that ClickHouse also misses a feature everyone in the community of users wish for, which is automatic regarding when you add a new node (sort of what elasticsearch does).

And before I forget, ClickHouse Kafka Engine is simply brilliant. The possibility of just publishing to Kafka and have your data not only inserted in your DB but also pre-processed is very powerful.

Let me know if I can help you with use cases we have.

Cheers

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#96

Post by srini20 »

sirffuzzylogik wrote: post #49
dominotw wrote: post #47

Earlier quoted context omitted.

> We are orders of magnitude faster than TimescaleDB and InfluxDB I think gp might be asking for a source for this claim. I see execution times on the demo but not sure if thats enough to say its faster than timescale.

j1897 is referring to https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-b...

(Hard to draw many meaningful conclusions from a single, extremely simple query without much explanation?)

Graph shows PostgreSQL as taking a long time, but doesn't say anything about configuration or parallelization. PostgreSQL should be able to parallelize that type of query since 9.6+, but I think they didn't use parallelization in these experiments with PostgreSQL, even though they used a bunch of parallel threads with QuestDB?

So would be good to know:

- What version of Postgres

- How many parallel workers for this query

- If employing JIT'ing the query

- If pre-warming the cache in PostgreSQL and configuring it to store fully in memory (as benchmarks with QuestDB appeared to do a two-pass to first mmap into memory, and only accounting for the second pass over in-memory data).

etc

Database benchmarking is pretty complex (and easy to bias), and most queries do not look like this toy one.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#99

Post by hawk_ »

mpsq wrote: post #32
shay_ker wrote: post #12

Absolutely love the story. TimescaleDB & InfluxDB have had a lot of posts on HN, so I'm sure others are wondering - how do we compare QuestDB to them? It sounds like performance is a big one, but I'm curious to hear your take on it.

As you said, performance is the main differentiator. We are orders of magnitude faster than TimescaleDB and InfluxDB on both data ingestion and querying. TimescaleDB relies on Postgres and has great SQL support. This is not the case for InfluxDB and this is where QuestDB shines: we do not plan to move away from SQL, we are very dedicated in bringing good support and some enhancements to make sure the querying languag…

do you do realtime steaming using SQL as well?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#100

Post by roskilli »

hintymad wrote: post #88

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless. A common flaw I see in many time-serie…

FYI M3 is now backed by M3DB, a distributed quorum read/write replicated time-series based columnar store specialized for realtime metrics. You can associate multiple values/timeseries with a single set of dimensions if you use Protobuf's to write data, for more see the storage engine documentation[0]. The current recommendation is not to limit your queries but limit the global data queried per second[1] by a single DB node by using a limit on the number of datapoints (inferred by blocks of datapoints per series). M3DB also uses an inverted index using FST segments that are mmap'd[2] similar to Apache Lucene and Elastic Search to make multi-dimensional searches on very large data sets fast (hundreds of trillions of datapoints, petabytes of data) which is a bit different to traditional columnar databases which focus on column stores and rarely is accompanied by a full text search inverted index.

[0]: https://docs.m3db.io/m3db/architecture/engine/

[1]: https://docs.m3db.io/operational_guide/resource_limits/

[2]: https://fosdem.org/2020/schedule/event/m3db/, https://fosdem.org/2020/schedule/event/m3db/attachments/audi... (PDF)

Post reply on HN