Am I the only one that's like "wtf is a time-series database compared to a normal one?"
Yes. Google it.
Launch HN: QuestDB (YC S20) – Fast open source time series database
21–30 of 173 posts
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#22Am I the only one that's like "wtf is a time-series database compared to a normal one?"
"normal" database does not preserve the order of data as it comes in. To get the data out you have to constantly rely on indexes or "order by" constantly to get chronological order back. Time series database should maintain the order of data and not rely on indexes to have data make sense again.
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#23Am I the only one that's like "wtf is a time-series database compared to a normal one?"
Here's a Medium post on Time Series Databases: https://medium.com/datadriveninvestor/what-are-time-series-d... But basically, they are databases where the time dimension is a critical aspect of the data. They handle never-ending streams of incoming time-stamped data. Think of streaming stock prices, or streaming temperature data from a sensor. You want to be able to query your data specifically in a time dimension -- let's see the temperature fluctuations over the past 24 hours, including the mean. Stuff like that.
A 'normal one' is typically used for things like transactional data which adds, deletes, and updates data among linked tables. While these transactions happen in time, the time component isn't necessarily a critical dimension of the data.
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#24This is great! Quick question: would you mind sharing why you went with Java vs something perhaps more performant like all C/C++ or Rust? I'd suspect language familiarity (which is 100% ok).
Java was the starting point. Back in the day Rust wasn't a thing and C++ projects were quite expensive to maintain. What Java does for us is IDE support, instant compilation time and super easy test coverage. For things that does require ultimate performance we do use C/C++ though. These libraries are packaged with Java and transparent to end user.
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#25Am I the only one that's like "wtf is a time-series database compared to a normal one?"
It organises storage so that operations like "drop all entries from before X", "get entries between X and Y with tags A or B" are cheap and so that storing ~linearly increasing values is super efficient with stupid ratios like 1:800+ for DoubleDelta.
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#26Good luck. I work on similar OS database engine for about decade now. It is not bad, but I think consulting is better way to get funds. Also avoid "zero gc", JVM can be surprisingly good.
Will be in touch :)
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#27Does postgres wire support mean QuestDB can be a drop-in replacement for a postgres database? Is this common?
QuestDB Head of DevRel here ... Yes, it can be a replacement of Postgres and it will be cheaper and faster. That being said, PGwire is still in alpha and is not 100% covered yet, so while migrating is possible, 100% Postgres Wire Protocol compatibility is not there yet.
For traditional transactional RDBMS data, I don't think it's a very common choice. For Time Series data, QuestDB is by far the fastest choice for Postgres-compatible SQL Time Series databases.
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#28mmap'd databases are really quick to implement. I implemented both row and column orientated databases. The traders and quants loved it - and adoption took off after we built a web interface that let you see a whole day and also zoom into exact trades with 100ms load times for even the most heavily traded symbols.
The benefits of mmaping and in general POSIX filesystem atomic properties are quick implementation, where you don't have to worry about buffer management. The filesystem and disk block remapping layer (in SSD or even HDDs now) are radically more efficient when data are given to them in contiguous large chunks. This is difficult to control with mmap where the OS may write out pages at its whim. However, even using advanced Linux system calls like mremap and fallocate, which try to improve the complexity of changing mappings and layout in the filesystem, eventually this lack of control over buffers will bite you.
And then when you look at it, the kernel (with help from the processor TLB) has to maintain complex data-structures to represent the mappings and their dirty/clean states. Accessing memory is not O(1) even when it is in RAM. Making something better tuned to a database than the kernel page management is a significant hurdle but that's where there are opportunities.
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#29Great story! Thanks for sharing
Re: Launch HN: QuestDB (YC S20) – Fast open source time series database
#30Testing out the demo:
SELECT * FROM trips WHERE tip_amount > 500 ORDER BY tip_amount DESC
Very interesting :-)