Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#81

Post by j1897 »

pachico wrote: post #78
neurostimulant wrote: post #67

Congrats! I've been looking for a time series database but most of them seems to be in-memory nosql databases. QuestDB might be exactly what I need. I'll definitely give it a try soon!

It would then be in your interest to know ClickHouse. I recommend your to have a look at it.

We've had one of their contributors bench questdb versus Clickhouse recently - you can find the results here https://github.com/questdb/questdb/issues/436

This came from a bench we had on our previous website versus them about summing 1 billion doubles.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#82

Post by airstrike »

Ixiaus wrote: post #21
thegreatpeter wrote: post #13

Am I the only one that's like "wtf is a time-series database compared to a normal one?"

Yes. Google it.

https://news.ycombinator.com/newsguidelines.html

Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#83

Post by sirffuzzylogik »

samsk wrote: post #77

Does it supports some kind of compression ? That's very important when storing billions of events.

Not yet but this is on the roadmap. In the meantime you could use a filesystem that supports compression such as ZFS or brtfs. The data is column orientated, this means that compression would be super efficient.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#84

Post by pachico »

I see this as a very interesting project. I use ClickHouse as OLAP and I'm very happy with it. I can tell you features that make me stick to it. If some day QuestDB offers them, I might explore the possibility to switch but never before. - very fast (I guess we're aligned here) - real time materialized views for aggregation functions (this is absolutely a killer feature that makes it quite pointless to be fast if you don't have it) - data warehouse features: I can join different data sources in one query. This allows me to join, for instance, my MySQL/MariaDB domain dB with it and produce very complete reports. - Grafana plugin - very easy to share/scale at table level - huge set of functions, from geo to URL, from ML to string manipulation - dictionaries: I can load maxdb geo dB and do real time localisation in queries I might add some more once they come to my mind. Having said this, good job!!!

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#86

Post by airstrike »

There's an opportunity for a tool that combines this sort of technology in the backend with a spreadsheet-like GUI powered by formulas and all the user friendliness that comes with a non-programmer interface. Wall Street would forever be changed. Source: I'm one of the poor souls fighting my CPU and RAM to do the same thing with Excel and non-native add-ins by {FactSet, Capital IQ, Bloomberg}

This stuff

    SELECT * FROM balances
    LATEST BY balance_ccy, cust_id
    WHERE timestamp
Makes me literally want to cry for knowing what is possible yet not being able to do this on my day job :(

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#87

Post by bluestreak »

airstrike wrote: post #86

There's an opportunity for a tool that combines this sort of technology in the backend with a spreadsheet-like GUI powered by formulas and all the user friendliness that comes with a non-programmer interface. Wall Street would forever be changed. Source: I'm one of the poor souls fighting my CPU and RAM to do the same thing with Excel and non-native add-ins by {FactSet, Capital IQ, Bloomberg} This stuff SELECT * FROM…

We are working on building a solid PostgreSQL support insofar as allowing ODBC driver to execute this type of query from Excel. This is work in progress with not that much left on it.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#88

Post by hintymad »

I'm curious how QuestDB handles dimensions. OLAP support with reasonably large number of dimensions and cardinality in the range of at least thousands is a must for modern-day time series database. Otherwise, what we get is only incremental improvement to Graphite -- a darling among startups, I understand, but a non-scalable extremely hard to use timeseries database nonetheless.

A common flaw I see in many time-series DBs is that they store one time series per combination of dimensions. As a result, any aggregation will result in scanning of potentially millions of time series. If any time-series DB claims that it is backed up by a key-value store, say, Cassandra, then the DB will have the aforementioned issue. For instance, Uber's M3 used to be backed up by Cassandra, and therefore would give this mysterious warning that an aggregation function exceeded the quota of 10,000 time series, even though from user's point of view the function dealt with a single time series with a number of dimensions.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#89

Post by bluestreak »

pachico wrote: post #84

I see this as a very interesting project. I use ClickHouse as OLAP and I'm very happy with it. I can tell you features that make me stick to it. If some day QuestDB offers them, I might explore the possibility to switch but never before. - very fast (I guess we're aligned here) - real time materialized views for aggregation functions (this is absolutely a killer feature that makes it quite pointless to be fast if you…

Thank you for the kind words and constructive feedback. We are here to build on feedback like this. Grafana plugin is coming soon.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#90

Post by fawce »

airstrike wrote: post #80
avthar wrote: post #31

Earlier quoted context omitted.

This is actually an underrated question. Time-series databases offer better performance and usability for dealing with time-series data (think DevOps metrics, data from IoT devices, stock prices etc, anything where you're monitoring and analyzing how things change over time) They allow you answer questions where time is the main component of interest much more quickly and easily: eg 1: IoT Sensors) Show me the averag…

> eg 2: Financial data) What's the price of stock X over the past 5 years This is so incredibly frustratingly slow to pull on FactSet and Capital IQ, it makes me want to pull my hair every time I have to build line charts over time for a period greater than 2 years

plug, but our system provides very fast access to price, fundamentals, estimates, etc: https://factset.quantopian.com
Post reply on HN