Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a Time Series Database from Scratch

fabxc.org

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Writing a Time Series Database from Scratch

#21

Post by weq »

rodionos wrote: post #8
bongonewhere wrote: post #5

Is like everyone creating a time series database from scratch?

The fact that many companies (FB, Uber, Google, Netflix, SO) roll their own TSDBs for metrics collection suggests that there is a real need. Or maybe there is not. It could a way to make boring system monitoring jobs fun and fancy again.

My first job was in 2005 writing a TSDB abstraction over sql server used for BI with defense.

we merged ~30Gb of data per night in about 8hrs on a 32core machine which was severly limited by a platter SAN. We would get about 1 disk failure /6months.

So do u need an actual TSDB? Or are other companies doing what we did?

Re: Writing a Time Series Database from Scratch

#22

Post by nunb »

zeptomu wrote: post #12
jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

I have to admit that I did not know time-series databases are a thing and just recently realized that because they came up more often. Unfortunately I do not have an answer for you, but I know traditional DB systems are hard to build, as we expect more reliability and guarantees from them compared to other (daemon) software. However I am interested to know why this kind of data and/or problems require specific softwa…

There is also timescale which is built atop Postgres. Or rather, modifies Postgres' query engine etc., this is not some rails app!

https://blog.timescale.com/when-boring-is-awesome-building-a...

Re: Writing a Time Series Database from Scratch

#23

Post by wickawic »

jandrewrogers wrote: post #15
jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

This observation applies to database engines generally. It is straightforward to explain: almost everyone with deep expertise in sophisticated database engine internals are contractually prohibited from disclosing anything about the design of such things. It is an industry steeped in trade secrets. Sophisticated database engines are littered with novel algorithms and designs that have never been published. Because ba…

> Sadly, academia is increasingly in the role of independently re-discovering what has been known for 10-20 years but treated as a secret.

This has been true of network programming for a long time as well. Private companies find novel ways of switching and routing packets, reducing latency, etc. and academia is left to pick up the pieces.

Re: Writing a Time Series Database from Scratch

#24

Post by shaklee3 »

jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

What about extremedb? It's not free, but I believe is on par. Or did you mean only free?

Re: Writing a Time Series Database from Scratch

#25

Post by stingraycharles »

jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

You should look into proprietary technologies, there are some very compelling offerings there. I have personally used QuasarDB ( https://www.quasardb.net/ ), and can say that it's on par with kdb+ in terms of performance, scales horizontally and has support for transactions and secondary indexes. They have a single node free edition as well.

Typically these solutions tend to follow the money, but if you ignore the list price, you can usually get a decent deal for a startup.

Re: Writing a Time Series Database from Scratch

#26

Post by sagichmal »

Rapzid wrote: post #11
bongonewhere wrote: post #5

Is like everyone creating a time series database from scratch?

I was under the impression that influxdb's storage engine was going to be a viable solution for Prometheus at some point. Now they are writing their own; not sure where that interest went.

That was never true AFAIK. I recall one of the core devs did a comparison of storage engines years ago and found Influx unsuitable; that work led to the current-gen storage engine.

Re: Writing a Time Series Database from Scratch

#27

Post by bbrazil »

sagichmal wrote: post #26
Rapzid wrote: post #11

Earlier quoted context omitted.

I was under the impression that influxdb's storage engine was going to be a viable solution for Prometheus at some point. Now they are writing their own; not sure where that interest went.

That was never true AFAIK. I recall one of the core devs did a comparison of storage engines years ago and found Influx unsuitable; that work led to the current-gen storage engine.

That was the 1st InfluxDB storage engine, things have evolved in the intervening years. The latest InfluxDB design is actually quite similar to the latest Prometheus design, what's different is our approaches to reliability and clustering.

It's presently looking like Influx might once again be an option for long term storage for Prometheus.

Re: Writing a Time Series Database from Scratch

#28

Post by bbrazil »

nickpeterson wrote: post #13
zeptomu wrote: post #12

Earlier quoted context omitted.

I have to admit that I did not know time-series databases are a thing and just recently realized that because they came up more often. Unfortunately I do not have an answer for you, but I know traditional DB systems are hard to build, as we expect more reliability and guarantees from them compared to other (daemon) software. However I am interested to know why this kind of data and/or problems require specific softwa…

I'm no expert, but I believe the Crux of the issue is how data is naturally organized and stored. In a row oriented database, most data is stored in pages that contain rows. There are often indexes with ordering, but unless a secondary index contains all the values needed (often called a covering index), the entire row must be retrieved to answer any query that uses that information. Most Time-series databases are co…

> Suddenly instead of loading a billion rows and averaging the value in one column, you're just accesing the column itself to answer the question. From an IO perspective that's a huge savings.

The other side of this is that writing out data in that form would naively be an iop per sample, as you're usually appending one sample to 100s of time series in one request.

A significant part of monitoring TSDB design is buffering up samples and batching writes in order reduce that iop rate to something sane.

For example in the right circumstances the 1.x Prometheus design can ingest 250k samples/s on a hard disk which provides ~100 iops/s.

Re: Writing a Time Series Database from Scratch

#29

Post by nicolaslem »

The description of this new storage engine does not explain how it manages the durability of the data.

When you compare with the extreme efforts traditional databases take to ensure that unplugging a server will never ever result in data loss[0], silencing this problem makes me wonder.

Is it that at this ingest rate even trying to ensure durability is a vain effort?

[0] https://www.sqlite.org/atomiccommit.html

Re: Writing a Time Series Database from Scratch

#30

Post by tveita »

nicolaslem wrote: post #29

The description of this new storage engine does not explain how it manages the durability of the data. When you compare with the extreme efforts traditional databases take to ensure that unplugging a server will never ever result in data loss[0], silencing this problem makes me wonder. Is it that at this ingest rate even trying to ensure durability is a vain effort? [0] https://www.sqlite.org/atomiccommit.html

They mention using a write-ahead-log, which should be sufficient for durability if implemented correctly.
Post reply on HN