Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a Time Series Database from Scratch

fabxc.org

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Writing a Time Series Database from Scratch

#31

Post by bbrazil »

nicolaslem wrote: post #29

The description of this new storage engine does not explain how it manages the durability of the data. When you compare with the extreme efforts traditional databases take to ensure that unplugging a server will never ever result in data loss[0], silencing this problem makes me wonder. Is it that at this ingest rate even trying to ensure durability is a vain effort? [0] https://www.sqlite.org/atomiccommit.html

Durability is not a requirement in that sense.

Consider that a regular scrape has happened and that data has been accepted by the DB but not yet flushed to disk.

Whether the database dies just before or just after the scrape produces the same result: The data for that scrape isn't present when the server restarts.

There plenty of other ways a scrape might not succeed that we have no control over (e.g. other end is overloaded, network blip), so there's not much point obsessing over this particular failure mode.

> Is it that at this ingest rate even trying to ensure durability is a vain effort?

It's not in vain, but it'd be a bad engineering tradeoff in terms of throughput.

Re: Writing a Time Series Database from Scratch

#32

Post by pas »

Rapzid wrote: post #11
bongonewhere wrote: post #5

Is like everyone creating a time series database from scratch?

I was under the impression that influxdb's storage engine was going to be a viable solution for Prometheus at some point. Now they are writing their own; not sure where that interest went.

That was about remote storage. And that was around the time when Influx itself used pluggable backends. Now influx has its own storage engine, and prometheus too.

Re: Writing a Time Series Database from Scratch

#33

Post by StreamBright »

jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

>> I still can't figure out why people can't even come close to KDB+

You can have a look what happend with extremely simple code bases over time. Imagine the same effect on larger code.

https://www.computer.org/cms/Computer.org/ComputingNow/issue...

Re: Writing a Time Series Database from Scratch

#34

Post by jnordwick »

shaklee3 wrote: post #24
jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

What about extremedb? It's not free, but I believe is on par. Or did you mean only free?

I was talking mostly about FOSS.

I've actually never heard of ExtremeDB. It's only in-memory though, so it can it even handle the huge datasets that timeseries/column dbs handle? I don't see any performance tests for it either.

Re: Writing a Time Series Database from Scratch

#35

Post by jnordwick »

stingraycharles wrote: post #25
jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

You should look into proprietary technologies, there are some very compelling offerings there. I have personally used QuasarDB ( https://www.quasardb.net/ ), and can say that it's on par with kdb+ in terms of performance, scales horizontally and has support for transactions and secondary indexes. They have a single node free edition as well. Typically these solutions tend to follow the money, but if you ignore the li…

I know there are other proprietary dbs out there. I was speaking of FOSS/open databases any cant they come even without an order of magnitude of commercial offerings. Quasar would be included in that commercial group.

I'm only slightly familiar with Quasar, but I believe it is just a key-value store with document characterists, not a general database. The comparisons on their website are mostly to other KV stores and document systems.

Re: Writing a Time Series Database from Scratch

#36

Post by zeveb »

jandrewrogers wrote: post #15
jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

This observation applies to database engines generally. It is straightforward to explain: almost everyone with deep expertise in sophisticated database engine internals are contractually prohibited from disclosing anything about the design of such things. It is an industry steeped in trade secrets. Sophisticated database engines are littered with novel algorithms and designs that have never been published. Because ba…

> This observation applies to database engines generally. It is straightforward to explain: almost everyone with deep expertise in sophisticated database engine internals are contractually prohibited from disclosing anything about the design of such things. It is an industry steeped in trade secrets. Sophisticated database engines are littered with novel algorithms and designs that have never been published. Because basic performance superiority is a key market differentiator, database companies have invested heavily in computer science R&D for decades to get an edge they are loath to share. Sadly, academia is increasingly in the role of independently re-discovering what has been known for 10-20 years but treated as a secret.

This makes an interesting case for software patents: what if database companies patented those insights instead of keeping them secret, and thus could keep others from using them for, say, 3-4 years? At the end of that time anyone could use those techniques.

This seems to me better than either trade secrets are the current absurdly-long patent lifetimes.

Re: Writing a Time Series Database from Scratch

#37

Post by Rapzid »

bbrazil wrote: post #27
sagichmal wrote: post #26

Earlier quoted context omitted.

That was never true AFAIK. I recall one of the core devs did a comparison of storage engines years ago and found Influx unsuitable; that work led to the current-gen storage engine.

That was the 1st InfluxDB storage engine, things have evolved in the intervening years. The latest InfluxDB design is actually quite similar to the latest Prometheus design, what's different is our approaches to reliability and clustering. It's presently looking like Influx might once again be an option for long term storage for Prometheus.

That was my impression after the influxdb storage engine rewrite; I'm guessing the similarities are not a coincidence :)

I'm not sure how interested influxdb would be to the idea, given their shrewd moves towards monetization, but it would be nice if the storage engine could be developed as a component of influxdb and adopted into Prometheus(ala rocks, level, etc).

Re: Writing a Time Series Database from Scratch

#38

Post by shaklee3 »

jnordwick wrote: post #34
shaklee3 wrote: post #24

Earlier quoted context omitted.

What about extremedb? It's not free, but I believe is on par. Or did you mean only free?

I was talking mostly about FOSS. I've actually never heard of ExtremeDB. It's only in-memory though, so it can it even handle the huge datasets that timeseries/column dbs handle? I don't see any performance tests for it either.

It does in-memory, hybrid, and disk. It's typically benchmarked versus kdb+ only:

http://financial.mcobject.com/downloads/white-papers-collate...

Re: Writing a Time Series Database from Scratch

#39

Post by geocar »

jandrewrogers wrote: post #15
jnordwick wrote: post #7

I still can't figure out why people can't even come close to KDB+. It is a real conundrum. I've been waiting patiently for something to show up, but the gap seems to keep getting bigger instead of smaller. Is it that people want to make the problem more complex that it needs to be? Is it that those who know most about these issues don't share their secrets so implemented from the outside often don't have a good under…

This observation applies to database engines generally. It is straightforward to explain: almost everyone with deep expertise in sophisticated database engine internals are contractually prohibited from disclosing anything about the design of such things. It is an industry steeped in trade secrets. Sophisticated database engines are littered with novel algorithms and designs that have never been published. Because ba…

KDB+ is not 20kLoC of dense C++.

Re: Writing a Time Series Database from Scratch

#40

Post by aksyn »

geocar wrote: post #39
jandrewrogers wrote: post #15

Earlier quoted context omitted.

This observation applies to database engines generally. It is straightforward to explain: almost everyone with deep expertise in sophisticated database engine internals are contractually prohibited from disclosing anything about the design of such things. It is an industry steeped in trade secrets. Sophisticated database engines are littered with novel algorithms and designs that have never been published. Because ba…

KDB+ is not 20kLoC of dense C++.

It's 1VLLoC of C, right? ;)
Post reply on HN