Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s About Time for Time Series Databases

nextplatform.com

Post reply on HN
Print view View original

141–150 of 151 posts

Re: It’s About Time for Time Series Databases

#141

Post by BillinghamJ »

cbcoutinho wrote: post #139

We use a proprietary database system that uses a 'flat-file' format (no idea what that means) and is primarily time series based due to the fact that we're logging sensor data. Since it's primarily a backend, you can't access it outside of their proprietary gui. It's also accessible as a linked server via SQL Server, but this is slow as hell for non-trivial queries. We use it within a power plant setting where we hea…

We have a similar setup for our telematics data (insurance company). It’s stored in files which contain DEFLATE’d BSON. Similarly, we have a custom GUI for viewing it, but do also have some terminal tools which can be used.

The solution to larger processing simply seems to be loading the files and batching them through whatever processing we want and storing the output in a proper database - which can then be queried instantly.

For a lot of time series data, this does seem like a pretty decent approach.

Re: It’s About Time for Time Series Databases

#142

Post by _0w8t »

dfox wrote: post #95
wenc wrote: post #85

Earlier quoted context omitted.

I looked into SQLite a while ago, and it wasn't so much its robustness that was the deal-breaker, but more the fact that it couldn't handle writes from parallel processes from multiple users (i.e. concurrency) like a normal SQL database could. I see SQLite as a file-format with a SQL interface.

There is Berkeley DB/SQL (now sadly owned by Oracle), which is essentially Sqlite with BDB instead it's original B-tree implementation, that supports multiple parallel writers. Also Firebird can be used as in-process embedded database and supports multiple writers (in fact in the server mode it uses same file-locking based synchronisation mechanisms as in embedded mode) On the other hand it is somewhat rare to see ap…

Berkley DB is not as reliable as SQLite. Subversion, before switching to own storage, where known for data corruption problems attributed to Berkley DB.

Re: It’s About Time for Time Series Databases

#144

Post by dustingetz »

dragandj wrote: post #86
dustingetz wrote: post #50

How about one made by Rich Hickey — https://www.datomic.com/

Sadly, Datomic lacks two of (arguably) very important features in this space: 1. It seems to not be optimized for speed (but it's difficult to say since the license forbids publishing benchmarks). 2. It's not open source.

I empathetically agree with #1 but object to #2. Datomic, on an architectural basis alone, is going to be strictly faster than say an RDBMS, because it can parallelize almost all of the workload. Read-side is entirely elastic; index and storage maintenance does not block the writer, and the single operation that must be serial is the actual ACID write primitive, the conditional-put to DynamoDB. In RDBMS, the ACID writer is competing for resources with all these other operations, because they can't be parallelized without sacrificing consistency.

Re: It’s About Time for Time Series Databases

#145

Post by amyjess »

_0w8t wrote: post #142
dfox wrote: post #95

Earlier quoted context omitted.

There is Berkeley DB/SQL (now sadly owned by Oracle), which is essentially Sqlite with BDB instead it's original B-tree implementation, that supports multiple parallel writers. Also Firebird can be used as in-process embedded database and supports multiple writers (in fact in the server mode it uses same file-locking based synchronisation mechanisms as in embedded mode) On the other hand it is somewhat rare to see ap…

Berkley DB is not as reliable as SQLite. Subversion, before switching to own storage, where known for data corruption problems attributed to Berkley DB.

I remember the old days when people would refer to berkdb as "Berserkley" because it tended to go berserk on your data. It's always had a bad reputation for reliability.

Re: It’s About Time for Time Series Databases

#146

Post by dragandj »

dustingetz wrote: post #144
dragandj wrote: post #86

Earlier quoted context omitted.

Sadly, Datomic lacks two of (arguably) very important features in this space: 1. It seems to not be optimized for speed (but it's difficult to say since the license forbids publishing benchmarks). 2. It's not open source.

I empathetically agree with #1 but object to #2. Datomic, on an architectural basis alone, is going to be strictly faster than say an RDBMS, because it can parallelize almost all of the workload. Read-side is entirely elastic; index and storage maintenance does not block the writer, and the single operation that must be serial is the actual ACID write primitive, the conditional-put to DynamoDB. In RDBMS, the ACID wri…

We are talking about time-series databases here. I guess they are more optimized for the specific workload than a generic RDBMS.

Re: It’s About Time for Time Series Databases

#147

Post by xapata »

wenc wrote: post #85
baldfat wrote: post #69

Earlier quoted context omitted.

> But try telling people you're using SQLite to store critical data... SQLite is solid and trusted by many but for some unknown reason some people refuse to see it as a solution for their needs. I have seen many databases that could easily been replaced by SQLite.

I looked into SQLite a while ago, and it wasn't so much its robustness that was the deal-breaker, but more the fact that it couldn't handle writes from parallel processes from multiple users (i.e. concurrency) like a normal SQL database could. I see SQLite as a file-format with a SQL interface.

It can handle some degree of concurrency via threading.

Re: It’s About Time for Time Series Databases

#148

Post by postwait »

kev009 wrote: post #114
acomjean wrote: post #76

I worked at a place that monitored power usage minute by minute across 1000s of locations. We just used MySQL with a time column. Maybe I'm not the target audience but I'm failing to see what this gets me. The problem is they say the data is imuatable and stored sequentially, allthough our data was imutable with devices on the net the data comes in random order when these ineviatably have connection problems. We alwa…

Then effectively your insert rate is between 16 and 60 per second, sure, you don't really need a sophisticated partitioning or log structured DB. Native static partitioning would give you a decent speedup without much thought. It's intro computer science, if you have a tree structure and fill it up, you spend a lot of time in the corners of theta notation. Timescale uses tightly integrated partitioning on the time ax…

Reducing "Other TSDBs" to log-structured-merge trees is misleading. Any large-scale TSDB has something sophisticated underneath and LSM is often just one tiny part of that. I would argue (as most do) that any TSDB "simply used an LSM" it would be doomed at any scale over time.

Re: It’s About Time for Time Series Databases

#149

Post by kev009 »

postwait wrote: post #148
kev009 wrote: post #114

Earlier quoted context omitted.

Then effectively your insert rate is between 16 and 60 per second, sure, you don't really need a sophisticated partitioning or log structured DB. Native static partitioning would give you a decent speedup without much thought. It's intro computer science, if you have a tree structure and fill it up, you spend a lot of time in the corners of theta notation. Timescale uses tightly integrated partitioning on the time ax…

Reducing "Other TSDBs" to log-structured-merge trees is misleading. Any large-scale TSDB has something sophisticated underneath and LSM is often just one tiny part of that. I would argue (as most do) that any TSDB "simply used an LSM" it would be doomed at any scale over time.

There was no reduction, it was intended as a pointer to one data structure some TSDBs are using underneath. I would bold and highlight the "etc" present there for you if the markup allowed it.

I hope a reader would become interested in what an LSM tree is (and perhaps as importantly isn't)

Re: It’s About Time for Time Series Databases

#150

Post by seangrogg »

j_s wrote: post #83
seangrogg wrote: post #79

Earlier quoted context omitted.

Source for origin? Legit interested.

https://en.wikipedia.org/wiki/SQLite#History > used aboard guided missile destroyers [...] operated without installing a database management system or requiring a database administrator PS. Thanks to your question, today I re-discovered https://unqlite.org , which may fulfill a requirement I've been facing for some time. Awesome!

Hey hey, unqlite looks pretty interesting. I'm both glad you were able to rediscover it and glad that I can have fun reading about it!
Post reply on HN