It’s About Time for Time Series Databases • Hacker News Board

Re: It’s About Time for Time Series Databases

#61

Post by stalller » Thu, Jan 25, 2018, 4:25 PM UTC

j_s wrote: post #41

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

There was a similar sub-discussion on the Datomic Cloud announcement last week: https://news.ycombinator.com/item?id=16168041#16168743 As an open source product it seems like TimescaleDB is less risky than Datomic. (Not that the comparison is even necessary -- I don't think they compete in the same category apart from both being non-relational DBs.) [edit:link fixed, thanks! (and thanks for taking the time to correct…

Hi, the link you posted seems broken. Also just wanted to clarify that TimescaleDB is in fact a relational database, being built as a PostgreSQL extension. Looking at the nature of time-series workloads (namely appending data about recent events) we can architect the database differently for time-series workloads and applications powered by time-series data; scaling insert rates and storage capacity, increasing query performance and simplifying operations, while retaining full-SQL and the reliability of Postgres.

Re: It’s About Time for Time Series Databases

#62

Post by creeble » Thu, Jan 25, 2018, 4:25 PM UTC

lbruder wrote: post #52

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

Simpler than a log file? What's the significant advantage of SQLite vs just appending to a file?

Re: It’s About Time for Time Series Databases

#63

Post by mfreed » Thu, Jan 25, 2018, 4:25 PM UTC

grzm wrote: post #49

amelius wrote: post #46

Earlier quoted context omitted.

Transactional support? I was under the impression that timeseries data is considered immutable.

Being transactional is orthogonal to being immutable. For example, if you want to ensure that a given set of writes is applied atomically, you would issue them in a transaction.

This is especially important if you are also dealing with metadata that you don't want to always denormalize into your time-series tables. If you don't insert it transactionally, you can also lose referential integrity between your time-series and relational (metadata) tables.

Re: It’s About Time for Time Series Databases

#64

Post by chrismccabe » Thu, Jan 25, 2018, 4:26 PM UTC

What is the benefits of timescaledb or influxdb? surely the enterprise pricing is going to be similar to KDB which has been around for years and a very mature product

Re: It’s About Time for Time Series Databases

#65

Post by PaulRobinson » Thu, Jan 25, 2018, 4:27 PM UTC

srcmap wrote: post #57

lbruder wrote: post #52

Earlier quoted context omitted.

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

Just tell them SQLite is used by ALL Android / iOS phones (billions of them) to store user's critical data. I have not heard of any cases of SQLite failure in phone, or anyone needing a DB admin to recover their phone contact data.

SQLite was built as a solution to the question "How do we put a SQL database on a nuclear submarine stealthily moving away from base for months at a time in secret, without putting a DBA in there with it?"

It can go wrong, it does go wrong, but it's a pretty solid solution, especially if you're not needing insane concurrency throughput.

The problem is many people need concurrency. They're running web apps over multiple servers. It's there where SQLite is a problem.

Re: It’s About Time for Time Series Databases

#66

Post by rpedela » Thu, Jan 25, 2018, 4:32 PM UTC

creeble wrote: post #62

lbruder wrote: post #52

Earlier quoted context omitted.

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

Simpler than a log file? What's the significant advantage of SQLite vs just appending to a file?

SQL query support?

Re: It’s About Time for Time Series Databases

#67

Post by cevian » Thu, Jan 25, 2018, 4:49 PM UTC

busterarm wrote: post #54

olympus wrote: post #47

The article recognizes that several time series databases already exist. They also say, "we aren't trying to compete against kdb+." They explain how they can handle time series data better than NoSQL databases that aren't time series focused. But what are they doing better than the existing time series databases? Surely they must have some advantage or they wouldn't have raised 16.1 million dollars.

Every time I read about some new solution to storing time series data, I always feel like I must be doing something wrong, but I've run into _zero_ problems yet. Every time I have to store time series data, I never really need ACID transactions. I definitely don't ever need to do updates or upserts. It's always write-once-read-many. ElasticSearch has always been the obvious choice to me and it has worked extremely we…

We've had some clients try this, find consistency issues between elastic nodes or postgres and elastic and start using TimescaleDB as a way to simplify the stack and application. But obviously YMMV and this is highly dependent on your query needs.

Re: It’s About Time for Time Series Databases

#68

Post by cevian » Thu, Jan 25, 2018, 4:54 PM UTC

amelius wrote: post #46

cevian wrote: post #28

Earlier quoted context omitted.

Yeah, to underscore some differences more completely, timescale has: - Full indexing and secondary index support. - Support for transactional semantics - Support for living along side relational data - including foreign keys to the relational data. - Full trigger and constraint support - Support for all of Postgres's native data types including JSON(B) and GIS location data.

Transactional support? I was under the impression that timeseries data is considered immutable.

That's not always true. There are tons of cases where time-series data gets corrected later. It is true that time-series is INSERT-mostly but UPDATES do happen.

Re: It’s About Time for Time Series Databases

#69

Post by baldfat » Thu, Jan 25, 2018, 5:02 PM UTC

lbruder wrote: post #52

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

> But try telling people you're using SQLite to store critical data...

SQLite is solid and trusted by many but for some unknown reason some people refuse to see it as a solution for their needs. I have seen many databases that could easily been replaced by SQLite.

Re: It’s About Time for Time Series Databases

#70

Post by scrollaway » Thu, Jan 25, 2018, 5:02 PM UTC

amyjess wrote: post #37

At my employer, we've recently (as of the middle of last year) been making a considerable effort to use InfluxDB to track our KPIs. It's working out wonderfully for us, and I'm expecting it'll get used more and more as the year goes on. What really floors me about Influx is how fast it is. A query that used to take hours in Oracle takes seconds in Influx. And the influx query is readable: rolling data up to various i…

I think you're saying a lot more about Oracle than about Influx :)

I also am using Influx at my company and the experience is not that great, mostly due to it being immature yes. For example, I currently have runaway disk usage and I have no way to know what table is using it. So I have to choose between losing data or continuously increasing storage.