Writing a Time Series Database from Scratch
1–10 of 40 posts
Re: Writing a Time Series Database from Scratch
#2You may also want to check https://github.com/criteo/biggraphite/wiki/BigGraphite-Annou... which is also about how to write a TSDB from Scatch but with different goals.
Re: Writing a Time Series Database from Scratch
#3I had a question about the following statement from the post:
>"Prometheus's storage layer has historically shown outstanding performance, where a single server is able to ingest up to one million samples per second as several million time series"
How are there one million samples per second equating to several million time series? Is a single sample not equivalent to a single data point in a time series db for a particular metric in Prometheus?
Re: Writing a Time Series Database from Scratch
#4I had a question about the following statement from the post: >"Prometheus's storage layer has historically shown outstanding performance, where a single server is able to ingest up to one million samples per second as several million time series" How are there one million samples per second equating to several million time series? Is a single sample not equivalent to a single data point in a time series db for a par…
I think this means that there are e.g. 10 million different time series, that each get a new sample appended every 10 seconds.
Re: Writing a Time Series Database from Scratch
#5Is like everyone creating a time series database from scratch?
Re: Writing a Time Series Database from Scratch
#6Exciting times in database land! It certainly seems like the good systems are converging on very similar storage architectures. This design is so similar to how Kafka and Kudu work internally.
As the raw storage seems pretty optimal now, I suspect next we'll see a comeback of indices for more precise queries to get another jump in performance.
Re: Writing a Time Series Database from Scratch
#7I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller.
Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good understanding of how to do things properly? If you were to asked the guy behind Prometheus if he's looked at the commercial offerings and what he's learned from them, would even be able to speak about them intelligently?
There seems to be a huge skills gap on these things that I can't put my finger on. I'd love to be able to use a real TSDB, even at only half the speed and usefulness. It would be great for these smaller firms that cant or wont pay the license fees for a commercial offering until they get larger.
Re: Writing a Time Series Database from Scratch
#8Is like everyone creating a time series database from scratch?
The fact that many companies (FB, Uber, Google, Netflix, SO) roll their own TSDBs for metrics collection suggests that there is a real need. Or maybe there is not. It could a way to make boring system monitoring jobs fun and fancy again.
Re: Writing a Time Series Database from Scratch
#9I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…
In-memory, K/Q, expensive, old, closed source?
Re: Writing a Time Series Database from Scratch
#10I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…
> It would be great for these smaller firms that cant or wont pay the license fees for a commercial offering until they get larger.
Why should we give that work away for free, especially if it's not that hard to roll your own for most small-scale needs?