Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
1–10 of 17 posts
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#2Hi, I'm Adam Cimarosti, one of the core engineers at QuestDB.
We built play.questdb.io to make it easy for anyone to try our database. No installation.
There's a Jupyter Lab notebook, data, sample code, queries and graphs.
We'd love to hear what you think.
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#3Hi, I'm Adam Cimarosti, one of the core engineers at QuestDB. We built play.questdb.io to make it easy for anyone to try our database. No installation. There's a Jupyter Lab notebook, data, sample code, queries and graphs. We'd love to hear what you think.
Reading your pitch here, i'd love to have a vague idea what questdb is and why I should care.
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#4Hi, I'm Adam Cimarosti, one of the core engineers at QuestDB. We built play.questdb.io to make it easy for anyone to try our database. No installation. There's a Jupyter Lab notebook, data, sample code, queries and graphs. We'd love to hear what you think.
Reading your pitch here, i'd love to have a vague idea what questdb is and why I should care.
I take it you did not visit the link?
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#5Hi, I'm Adam Cimarosti, one of the core engineers at QuestDB. We built play.questdb.io to make it easy for anyone to try our database. No installation. There's a Jupyter Lab notebook, data, sample code, queries and graphs. We'd love to hear what you think.
Reading your pitch here, i'd love to have a vague idea what questdb is and why I should care.
Most databases store the latest state of something.
We don't. We ingest events.
After all, life is a function of time :-)
The whole world ticks and we take those ticks and store them.
If part of your application tracks anything happening over time (trades, ocean pollution levels, ships moving, rocket simulation metrics.. or whatever else then it makes sense to store those events in a time series database.
What we provide, primarily, is two basic pieces of functionality:
(1) Taking in lots of events FAST. Our ingestion rate is high (and we also integrate with things like Kafka, Pandas -- see the notebook, etc). Each of our time series tables (we support regular ones too) comes with a special timestamp column.
(2) Specialized SQL to make sense of data that's changed over time, such as grouping and resampling by time and more. Take a look at our docs for things like SAMPLE BY, LATEST ON, ASOF JOIN, LT JOIN and more. On disk, we also guarantee that all records are sorted by time and this gives us great query performance for these time-based types of queries.
PS. We're also wire-compatible with PostgreSQL.
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#6Earlier quoted context omitted.
Reading your pitch here, i'd love to have a vague idea what questdb is and why I should care.
Most databases store the latest state of something. We don't. We ingest events. After all, life is a function of time :-) The whole world ticks and we take those ticks and store them. If part of your application tracks anything happening over time (trades, ocean pollution levels, ships moving, rocket simulation metrics.. or whatever else then it makes sense to store those events in a time series database. What we pro…
So I guess it would be fair to say you compete with Timescale and Clickhouse as a timeseries database?
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#7Earlier quoted context omitted.
Most databases store the latest state of something. We don't. We ingest events. After all, life is a function of time :-) The whole world ticks and we take those ticks and store them. If part of your application tracks anything happening over time (trades, ocean pollution levels, ships moving, rocket simulation metrics.. or whatever else then it makes sense to store those events in a time series database. What we pro…
So I guess it would be fair to say you compete with Timescale and Clickhouse as a timeseries database?
yes correct - although Clickhouse is more of an OLAP database. Timescale is built on top of Postgres, while QuestDB is built from scratch with Postgres wire compatibility. You can run benchmarks on https://github.com/timescale/tsbs
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#8Earlier quoted context omitted.
Reading your pitch here, i'd love to have a vague idea what questdb is and why I should care.
Most databases store the latest state of something. We don't. We ingest events. After all, life is a function of time :-) The whole world ticks and we take those ticks and store them. If part of your application tracks anything happening over time (trades, ocean pollution levels, ships moving, rocket simulation metrics.. or whatever else then it makes sense to store those events in a time series database. What we pro…
I was once in the market for time series databases, but all I could find required down sampling of older data. I don't know if this has changed, and to be fair I haven't been looking for quite some time, but does yours allow for keeping data with the captured precision in perpetuity (or until my hard drive fills up)? My guess from the way you describe your approach is yes, but I wanted to check.
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#9Earlier quoted context omitted.
Most databases store the latest state of something. We don't. We ingest events. After all, life is a function of time :-) The whole world ticks and we take those ticks and store them. If part of your application tracks anything happening over time (trades, ocean pollution levels, ships moving, rocket simulation metrics.. or whatever else then it makes sense to store those events in a time series database. What we pro…
I was once in the market for time series databases, but all I could find required down sampling of older data. I don't know if this has changed, and to be fair I haven't been looking for quite some time, but does yours allow for keeping data with the captured precision in perpetuity (or until my hard drive fills up)? My guess from the way you describe your approach is yes, but I wanted to check.
Yes. We're pretty good with large volumes of data.
Eventually all local drives fill up though.
When ingesting data we partition data by time. By default we partition by day. This give you the flexibility to detach partitions, store them somewhere slower and cheaper with more capacity for longer term storage and reattach them later if need be.
Built on top of our open source primary product, we also have a cloud variant of QuestDB which runs on AWS. One of the things that we're building there is cold storage. It will automate this process onto S3 such that if a query ever needs to access this older data it will re-enstate it automatically for you with no admin overhead.
Re: Show HN: QuestDB with Python, Pandas and SQL in a Jupyter notebook – no install
#10Hi, I'm Adam Cimarosti, one of the core engineers at QuestDB. We built play.questdb.io to make it easy for anyone to try our database. No installation. There's a Jupyter Lab notebook, data, sample code, queries and graphs. We'd love to hear what you think.
This is really cool -- congrats on the launch! Similarly, the team over at AuthZed has created a playground for SpiceDB[0], by using WebAssembly and Monaco.
We debated for hours whether or not to go the notebook route. I'm sure y'all did something similar; would you care to share your reasons for going with the notebook?