Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s About Time for Time Series Databases

nextplatform.com

Post reply on HN
Print view View original

111–120 of 151 posts

Re: It’s About Time for Time Series Databases

#111

Post by manigandham »

lima wrote: post #94

ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database. There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to. It's the most promising development in this space I've seen in a long time. https://clickhouse.yandex/ https://clickhouse.yandex/docs/en/table_engines/graphitemerg...

I'd also recommend Druid, MemSQL, SnappyData, MapD and other column-oriented databases. Any of them can partition on a time column with full SQL and extremely fast aggregations and high compression that come from columnar storage.

Re: It’s About Time for Time Series Databases

#112

Post by tzakrajs »

amyjess wrote: post #37

At my employer, we've recently (as of the middle of last year) been making a considerable effort to use InfluxDB to track our KPIs. It's working out wonderfully for us, and I'm expecting it'll get used more and more as the year goes on. What really floors me about Influx is how fast it is. A query that used to take hours in Oracle takes seconds in Influx. And the influx query is readable: rolling data up to various i…

Apache Hive is also a great piece of software that I would recommend for these use cases.

Re: It’s About Time for Time Series Databases

#113

Post by j_s »

tzakrajs wrote: post #110
j_s wrote: post #58

Earlier quoted context omitted.

Yes, collecting/aggregating for querying seems like the sticking point.

What does collecting or aggregating have to do with replication/HA?

It was my improvisational approach to pointing in the right direction. https://en.wikipedia.org/wiki/Yes,_and....

I don't really see how replication/HA is an issue in this setup beyond normal file system replication/HA (which seems to be a solved "spend money on NAS" problem in the enterprise world). If your filesystem isn't reliable / delivering what's needed then you're gonna have a bad time no matter the storage approach; I doubt some unusual option is involved though it would be cool to be proven wrong.

The only time the filesystem could even potentially have read/write conflicts (assuming 1-to-1 correlation between devices writing to files) is when querying. I did wind up muddling the terminology in the interest of jumping to the actual problem.

Re: It’s About Time for Time Series Databases

#114

Post by kev009 »

acomjean wrote: post #76

I worked at a place that monitored power usage minute by minute across 1000s of locations. We just used MySQL with a time column. Maybe I'm not the target audience but I'm failing to see what this gets me. The problem is they say the data is imuatable and stored sequentially, allthough our data was imutable with devices on the net the data comes in random order when these ineviatably have connection problems. We alwa…

Then effectively your insert rate is between 16 and 60 per second, sure, you don't really need a sophisticated partitioning or log structured DB. Native static partitioning would give you a decent speedup without much thought.

It's intro computer science, if you have a tree structure and fill it up, you spend a lot of time in the corners of theta notation. Timescale uses tightly integrated partitioning on the time axis to deal with write performance and aging data out. Other popular TSDBs are plays on log structured merge trees etc

Re: It’s About Time for Time Series Databases

#115

Post by pas »

kodablah wrote: post #14
walterbell wrote: post #6

Earlier quoted context omitted.

Is the concern mitigated by this being an open-source Postgres extension? One question would be whether they have attracted external OSS contributors.

Depends on your reliance I suppose. Appears most of TimescaleDB's vendor-specific pieces are in the DDL area, but there are plenty of DML-specific extensions. I'm not saying TimescaleDB might not get to a level of maturity where I'd use it, I personally just can't justify it today. I've found DBs and ops/orchestration engines to be the most painful to move off from if you need to.

They also provide the most bang for the feature-complexity buck. (A DB is something you don't want to roll on your own, and you want to use the purpose-built one for your use case. A columnar store for time series, a scatter-gather SSD + random read optimized engine for hot OLTP, a fancy multi level cache/tier BLOB store for photos/files, CockroachDB for geo consistency, etcd for local consistent config, and so on.)

Re: It’s About Time for Time Series Databases

#116

Post by j4ship »

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

those are concerns of every project , we will see if timescale is different

But is it good enough , as a product it needs to prove to me its good enough to switch. Many people dont need a specific time database. Many people have created solutions on top of their current database.

We have hadoop and other post processors that can allow us to off load old data but raise features to the user to get data in the past.

Is this better? And how much more does it cost? Oracle didnt get successful becuase people couldnt store data before them. They were successful becuase it became a need: You have a DB solution or you dont and if you dont you cant be secure in your "data assets" (Martin Fowler talks about this).

So if I dont have a Time database ... what cant I do? Who is going to not do business with me unless I have a time db? I hope you have some good enterprise friends.

Re: It’s About Time for Time Series Databases

#117

Post by da_chicken »

baldfat wrote: post #108
da_chicken wrote: post #102

Earlier quoted context omitted.

Yep. Parallel performance is the biggest issue I've seen with SQLite. SpiceWorks is the biggest example of a product that doesn't scale because they've elected to use SQLite (and only SQLite). There's no hard limit on the number of devices, but most people say not to exceed about 1,000 devices. We tested it out against 800 devices. We started it on Friday. On Monday, it was still running. The only other issue I've se…

There are plenty of problems that don't need parallel performance.

And there are plenty that do. What's your point?

Re: It’s About Time for Time Series Databases

#118

Post by amyjess »

scrollaway wrote: post #70
amyjess wrote: post #37

At my employer, we've recently (as of the middle of last year) been making a considerable effort to use InfluxDB to track our KPIs. It's working out wonderfully for us, and I'm expecting it'll get used more and more as the year goes on. What really floors me about Influx is how fast it is. A query that used to take hours in Oracle takes seconds in Influx. And the influx query is readable: rolling data up to various i…

I think you're saying a lot more about Oracle than about Influx :) I also am using Influx at my company and the experience is not that great, mostly due to it being immature yes. For example, I currently have runaway disk usage and I have no way to know what table is using it. So I have to choose between losing data or continuously increasing storage.

One bit of immaturity I just remembered: InfluxDB does a terrible job of telling you when you've fat-fingered a query. You can select from a series that doesn't exist or reference a tag that doesn't exist inside a 'where' clause, and InfluxDB will happily return 0 results instead of giving an error.

I've spent some time trying to figure out why I'm not getting data, and it turns out that it was because I typoed the name of the series.

I still like Influx though, warts and all.

Re: It’s About Time for Time Series Databases

#119

Post by cevian »

manigandham wrote: post #111
lima wrote: post #94

ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database. There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to. It's the most promising development in this space I've seen in a long time. https://clickhouse.yandex/ https://clickhouse.yandex/docs/en/table_engines/graphitemerg...

I'd also recommend Druid, MemSQL, SnappyData, MapD and other column-oriented databases. Any of them can partition on a time column with full SQL and extremely fast aggregations and high compression that come from columnar storage.

Hi, you've posted this notion that partitioning a column store by time would yield the same result as TimescaleDB a few times, so thought we'd jump in and clear things up. We fully agree that column stores have their place, particularly if you have a massive number of metrics, and all you care are roll-ups on single column axes. There are some major differences between TimescaleDB and column stores. Namely, TimescaleDB supports a lot of features that column stores in general do not.

- Secondary Indexes.

- Transactional semantics.

- Can operate on data sets greater than available memory (doesn't have to be all in-memory unlike memSQL and some others). Time-series data is voluminous.

- A whole bunch of specialized time-based optimizations that optimize query plans when working with time-based indexes and data.

- Constraints - Including foreign keys.

- Triggers

- Joins with relational data.

- Full SQL - allowing you to use complex queries and window functions

- Compatible with data tools that use SQL - which gets you gets you the richest ecosystem of tools in the data world

- The full gamut of Postgres datatypes including JSON/B and GIS location data

- 20+ years of reliability, tested backups, live streaming replication, etc.

- Geospatial support through best-in-class PostGIS

And of course, we're only getting started :)

Re: It’s About Time for Time Series Databases

#120

Post by cevian »

lima wrote: post #94

ClickHouse (the analytics DMBS by Yandex), while not explicitly designed as such, is a fantastic time series database. There's even a special backend, the GraphiteMergeTree, which does staggered downsampling, something most TSDBs aren't able to. It's the most promising development in this space I've seen in a long time. https://clickhouse.yandex/ https://clickhouse.yandex/docs/en/table_engines/graphitemerg...

Clickhouse is very cool. But note that it does not support transactional and relational semantics and does not have real-time updates or deletes. Thus, its meant for very different applications than TimescaleDB. I would classify Clickhouse more in the data-warehouse space...
Post reply on HN