Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: QuestDB (YC S20) – Fast open source time series database

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 173 posts

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#41

Post by j1897 »

avthar wrote: post #36
mpsq wrote: post #32

Earlier quoted context omitted.

As you said, performance is the main differentiator. We are orders of magnitude faster than TimescaleDB and InfluxDB on both data ingestion and querying. TimescaleDB relies on Postgres and has great SQL support. This is not the case for InfluxDB and this is where QuestDB shines: we do not plan to move away from SQL, we are very dedicated in bringing good support and some enhancements to make sure the querying languag…

Are there any performance comparisons to TimescaleDB and Influx that you can share? A blog post perhaps?

hi there - co-founder of questdb here. The demo on our website hosts a 1.6 billion rows NYC taxi dataset with 10 years of weather data with around 30-minute resolution and weekly gas prices over the last decade.

We've got example of queries in the demo, and you can see the execution times there.

We have posted a blog post comparing the ingestion speed of InfluxDB and QuestDB via InfluxDB Line Protocol some time ago: https://questdb.io/blog/2019/12/19/lineprot

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#42

Post by bluestreak »

vii wrote: post #28

mmap'd databases are really quick to implement. I implemented both row and column orientated databases. The traders and quants loved it - and adoption took off after we built a web interface that let you see a whole day and also zoom into exact trades with 100ms load times for even the most heavily traded symbols. The benefits of mmaping and in general POSIX filesystem atomic properties are quick implementation, wher…

thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places where we could not beat aggregation of julia, but the delta wasn't very big. This could have been down to mapped memory. We will definitely try things with direct memory too!

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#45

Post by aloukissas »

bluestreak wrote: post #24
aloukissas wrote: post #4

This is great! Quick question: would you mind sharing why you went with Java vs something perhaps more performant like all C/C++ or Rust? I'd suspect language familiarity (which is 100% ok).

Java was the starting point. Back in the day Rust wasn't a thing and C++ projects were quite expensive to maintain. What Java does for us is IDE support, instant compilation time and super easy test coverage. For things that does require ultimate performance we do use C/C++ though. These libraries are packaged with Java and transparent to end user.

Makes sense, that's what I also guessed.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#46

Post by judofyr »

Congratulations on launching! It looks like a great product. Some technical questions which I didn’t see answered on my first glance:

(1) Is it a single-server only, or is it possible to store data replicated as well?

(2) I’m guessing that all the benchmarks were done with all the hot data paged into memory (correct?); what’s the performance once you hit the disk? How much memory do you recommend running with?

(3) How’s the durability? How often do you write to disk? How do you take backups? Do you support streaming backups? How fast/slow/big are snapshot backups?

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#47

Post by dominotw »

j1897 wrote: post #41
avthar wrote: post #36

Earlier quoted context omitted.

Are there any performance comparisons to TimescaleDB and Influx that you can share? A blog post perhaps?

hi there - co-founder of questdb here. The demo on our website hosts a 1.6 billion rows NYC taxi dataset with 10 years of weather data with around 30-minute resolution and weekly gas prices over the last decade. We've got example of queries in the demo, and you can see the execution times there. We have posted a blog post comparing the ingestion speed of InfluxDB and QuestDB via InfluxDB Line Protocol some time ago:…

> We are orders of magnitude faster than TimescaleDB and InfluxDB

I think gp might be asking for a source for this claim.

I see execution times on the demo but not sure if thats enough to say its faster than timescale.

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#48

Post by joan4 »

jaydub wrote: post #38
santafen wrote: post #27

Earlier quoted context omitted.

QuestDB Head of DevRel here ... Yes, it can be a replacement of Postgres and it will be cheaper and faster. That being said, PGwire is still in alpha and is not 100% covered yet, so while migrating is possible, 100% Postgres Wire Protocol compatibility is not there yet. For traditional transactional RDBMS data, I don't think it's a very common choice. For Time Series data, QuestDB is by far the fastest choice for Pos…

Yeah, I checked it out and wanted to use but a bunch of regular old SQL queries don't work. Please add support for the old fashioned group by syntax! (This will be helpful for getting to a true drop-in replacement!)

QuestDB dev here. We added support for GROUP BY syntax in yesterday's release

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#49

Post by sirffuzzylogik »

dominotw wrote: post #47
j1897 wrote: post #41

Earlier quoted context omitted.

hi there - co-founder of questdb here. The demo on our website hosts a 1.6 billion rows NYC taxi dataset with 10 years of weather data with around 30-minute resolution and weekly gas prices over the last decade. We've got example of queries in the demo, and you can see the execution times there. We have posted a blog post comparing the ingestion speed of InfluxDB and QuestDB via InfluxDB Line Protocol some time ago:…

> We are orders of magnitude faster than TimescaleDB and InfluxDB I think gp might be asking for a source for this claim. I see execution times on the demo but not sure if thats enough to say its faster than timescale.

j1897 is referring to https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-b...

Re: Launch HN: QuestDB (YC S20) – Fast open source time series database

#50

Post by bluestreak »

judofyr wrote: post #46

Congratulations on launching! It looks like a great product. Some technical questions which I didn’t see answered on my first glance: (1) Is it a single-server only, or is it possible to store data replicated as well? (2) I’m guessing that all the benchmarks were done with all the hot data paged into memory (correct?); what’s the performance once you hit the disk? How much memory do you recommend running with? (3) Ho…

thank you!

- replication is in the works, this is going to be both TCP and UDP based, column-first, very fast.

- yes, benchmarks are indeed are done on second pass over the mmaped pages. First pass would trigger IO, which is OS-driven and dependant on disk speed. We've seen well over 1.5Gb/s on disks that support this speed. Columns are mapped into memory separately and they are lazy accessed. So the memory footprint depends on what data your SQLs actually lift. We go quite far to minimize false disk reads by working with rowids as much and possible. For example 'order by' will need memory for 8 x row_count bytes in most cases.

- durability is something we want user to have control over. Under the hood we have these commit modes:

https://github.com/questdb/questdb/blob/master/core/src/main...

NOSYNC = means OS flushes memory whenever. That said, we use sliding 16MB memory window when writing. Flushes will trigger by unmapping pages. ASYNC = we call msync(async) SYNC = we call msync(sync)

Post reply on HN