Isn't it a bit weird to launch a "never delete anything" store today and not mention anything about European GDPR requirements?
How would you go about deleting a users data upon request?
It’s About Time for Time Series Databases
91–100 of 151 posts
Re: It’s About Time for Time Series Databases
#92> nobody wants to have large grain snapshots of data for any dataset that is actually comprised of a continuous stream of data points Except, of course, for those who realize that the precision of a statistic only increases at sqrt(n) and that a biased dataset will remain biased regardless of how much data you have. I'll take a large grain dataset that I can load on my computer and analyze in five minutes over a fine…
There are a lot of use cases (for example security) where it is not the case.
Re: It’s About Time for Time Series Databases
#93Isn't it a bit weird to launch a "never delete anything" store today and not mention anything about European GDPR requirements? How would you go about deleting a users data upon request?
You delete it. It's still a database. What's being done here is just smart partition management as an extension to Postgres, similar to CitusDB.
You can also use any column-oriented relational database with a time-based partition key and do the same thing.
Re: It’s About Time for Time Series Databases
#94ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database.
There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to.
It's the most promising development in this space I've seen in a long time.
https://clickhouse.yandex/docs/en/table_engines/graphitemerg...
Re: It’s About Time for Time Series Databases
#95Earlier quoted context omitted.
> But try telling people you're using SQLite to store critical data... SQLite is solid and trusted by many but for some unknown reason some people refuse to see it as a solution for their needs. I have seen many databases that could easily been replaced by SQLite.
I looked into SQLite a while ago, and it wasn't so much its robustness that was the deal-breaker, but more the fact that it couldn't handle writes from parallel processes from multiple users (i.e. concurrency) like a normal SQL database could. I see SQLite as a file-format with a SQL interface.
There is Berkeley DB/SQL (now sadly owned by Oracle), which is essentially Sqlite with BDB instead it's original B-tree implementation, that supports multiple parallel writers. Also Firebird can be used as in-process embedded database and supports multiple writers (in fact in the server mode it uses same file-locking based synchronisation mechanisms as in embedded mode)
On the other hand it is somewhat rare to see aplication that really needs multiple writers and would not be better served by full fledged database server.
Re: It’s About Time for Time Series Databases
#96Earlier quoted context omitted.
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
Simpler than a log file? What's the significant advantage of SQLite vs just appending to a file?
On disk support?
Re: It’s About Time for Time Series Databases
#97Earlier quoted context omitted.
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
That's a good pattern for straight data retrieval. Unfortunately if you need to do aggregated queries on all of the SQLLite tables, things may be challenging. But if you could somehow connect Spark to a folder (on a distributed FS) of these SQLite files... Edit: Also SQLite has a limitation that only one process can write to it at a given a time. For this particular use case though, it shouldn't be a problem unless y…
I mean, depends on the aggregation you need imo. Shouldn't be too hard (tm) to rig up some distributed query pipeline. (as long as you are ok with coding per query, instead of the convenience of sql)
Re: It’s About Time for Time Series Databases
#98ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database. There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to. It's the most promising development in this space I've seen in a long time. https://clickhouse.yandex/ https://clickhouse.yandex/docs/en/table_engines/graphitemerg...
Direct link how to use as a graphite whisper replacement https://github.com/yandex/graphouse
Telegram channel https://t.me/clickhouse_en
Re: It’s About Time for Time Series Databases
#99ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database. There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to. It's the most promising development in this space I've seen in a long time. https://clickhouse.yandex/ https://clickhouse.yandex/docs/en/table_engines/graphitemerg...
Completely unrelated question, but how on earth is 'clickhouse.yandex' a valid web address?
Re: It’s About Time for Time Series Databases
#100ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database. There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to. It's the most promising development in this space I've seen in a long time. https://clickhouse.yandex/ https://clickhouse.yandex/docs/en/table_engines/graphitemerg...
Completely unrelated question, but how on earth is 'clickhouse.yandex' a valid web address?
The same way https://domains.google is, you pay - you get your own TLD