Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s About Time for Time Series Databases

nextplatform.com

Post reply on HN
Print view View original

81–90 of 151 posts

Re: It’s About Time for Time Series Databases

#81

Post by akulkarni »

RobAtticus wrote: post #19
arbesfeld wrote: post #3

We use Cassandra extensively at https://logrocket.com . How does performance compare vs Cassandra when you don't need the semantics or transactions of SQL? I'm surprised the article only provides benchmarks against PostgreSQL.

That is currently next in our pipeline for a benchmark blog post. Early results look good on both the read and write side in terms of raw performance, with the benefit of more complex queries being more easily expressable.

To add to this: We've found that the benefit of Cassandra's approach is when scaling out to 100s of nodes (which TimescaleDB currently does not support). But we found Cassandra's per node usage not that impressive. We'll have more benchmarks vs Cassandra soon (@robatticus is literally the person working on it), but suffice to say that users have already replaced multi-node Cassandra clusters (e.g., 10 nodes) with a single TimescaleDB instance (optionally adding read-only replicas for hot standbys).

Re: It’s About Time for Time Series Databases

#82

Post by akulkarni »

tw1010 wrote: post #17

I have a hard time shaking the feeling that database innovation is being forced not by necessity, but by a culture of blind relentless change.

It (and tech innovation in general) is also driven by people scratching their own itch. That's where we started. And we actually didn't start as a DB company, but as an IoT company storing lots of sensor data. We needed a certain kind of time-series database, tried several options, but nothing worked. So we built our own, and then realized other people needed it as well.

And we picked Postgres as our starting point exactly because it wasn't new and shiny but boring and it worked.

More on this here [1] if you're interested.

[1] https://blog.timescale.com/when-boring-is-awesome-building-a...

Re: It’s About Time for Time Series Databases

#83

Post by j_s »

seangrogg wrote: post #79
PaulRobinson wrote: post #65

Earlier quoted context omitted.

SQLite was built as a solution to the question "How do we put a SQL database on a nuclear submarine stealthily moving away from base for months at a time in secret, without putting a DBA in there with it?" It can go wrong, it does go wrong, but it's a pretty solid solution, especially if you're not needing insane concurrency throughput. The problem is many people need concurrency. They're running web apps over multip…

Source for origin? Legit interested.

https://en.wikipedia.org/wiki/SQLite#History

>used aboard guided missile destroyers [...] operated without installing a database management system or requiring a database administrator

PS. Thanks to your question, today I re-discovered https://unqlite.org, which may fulfill a requirement I've been facing for some time. Awesome!

Re: It’s About Time for Time Series Databases

#84

Post by camel_gopher »

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

Data portability is certainly an important issue here to consider here given the points you mentioned. When adopting a vendor, the relevant question is "How much work is it to get data in?". And when planning for the contingency of vendor loss, the question is "How much work is it to get the data out?".

The first question is tied closely to the interfaces provided by the database, and those interfaces are usually commensurate with the associated tooling. In the case of time series databases, they include statsd, collectd, and good old JSON. Relational databases utilize SQL as expected. Take the example of migrating from Oracle to Postgres; you shouldn't have to spend an inordinate amount of retooling your applications because the data interfaces are similar.

Data egress is similar; SQL databases generally provide SQL/CSV exports, and most TSDBs (the good ones at least) allow you to get the data out via JSON through a REST interface.

Whether or not the DB you choose is open source, or from a small vendor, you should pick the right tool to solve your problem with data portability in mind. (disclosure, I work for http://irondb.io, and have used Postgres for 18 years)

Re: It’s About Time for Time Series Databases

#85

Post by wenc »

baldfat wrote: post #69
lbruder wrote: post #52

Earlier quoted context omitted.

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

> But try telling people you're using SQLite to store critical data... SQLite is solid and trusted by many but for some unknown reason some people refuse to see it as a solution for their needs. I have seen many databases that could easily been replaced by SQLite.

I looked into SQLite a while ago, and it wasn't so much its robustness that was the deal-breaker, but more the fact that it couldn't handle writes from parallel processes from multiple users (i.e. concurrency) like a normal SQL database could.

I see SQLite as a file-format with a SQL interface.

Re: It’s About Time for Time Series Databases

#86

Post by dragandj »

dustingetz wrote: post #50

How about one made by Rich Hickey — https://www.datomic.com/

Sadly, Datomic lacks two of (arguably) very important features in this space:

1. It seems to not be optimized for speed (but it's difficult to say since the license forbids publishing benchmarks).

2. It's not open source.

Re: It’s About Time for Time Series Databases

#87

Post by dominotw »

baldfat wrote: post #69
lbruder wrote: post #52

Earlier quoted context omitted.

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

> But try telling people you're using SQLite to store critical data... SQLite is solid and trusted by many but for some unknown reason some people refuse to see it as a solution for their needs. I have seen many databases that could easily been replaced by SQLite.

probably cos of word "Lite" in their name.

Re: It’s About Time for Time Series Databases

#88

Post by pfranz »

busterarm wrote: post #54
olympus wrote: post #47

The article recognizes that several time series databases already exist. They also say, "we aren't trying to compete against kdb+." They explain how they can handle time series data better than NoSQL databases that aren't time series focused. But what are they doing better than the existing time series databases? Surely they must have some advantage or they wouldn't have raised 16.1 million dollars.

Every time I read about some new solution to storing time series data, I always feel like I must be doing something wrong, but I've run into _zero_ problems yet. Every time I have to store time series data, I never really need ACID transactions. I definitely don't ever need to do updates or upserts. It's always write-once-read-many. ElasticSearch has always been the obvious choice to me and it has worked extremely we…

I had set up ElasticSearch for logging events and was thinking about using it to store metrics as well. It seemed like it ticked all the boxes and would work pretty well when I was messing around with it.

I ended up leaving that job and set up something specifically for metrics from scratch. I didn't compare 1 to 1, but this was much faster to query and had much lower requirements (disk space, memory, etc).

Both use-cases are using it as a time-series database, and there's no reason ElasticSearch couldn't work for both of those use-cases for many people. When using two different backends (one for events and another for metrics) I meant drawing a line when creating/logging data and two ways to query...which sucks for training. You also would have to maintain/archive two different data stores.

Re: It’s About Time for Time Series Databases

#89

Post by Krunkel »

amyjess wrote: post #37

At my employer, we've recently (as of the middle of last year) been making a considerable effort to use InfluxDB to track our KPIs. It's working out wonderfully for us, and I'm expecting it'll get used more and more as the year goes on. What really floors me about Influx is how fast it is. A query that used to take hours in Oracle takes seconds in Influx. And the influx query is readable: rolling data up to various i…

I can attest to how great InfluxDB is. Have used it in a few applications that rely on time based data and I am constantly blown away by how fast the query times are.

Re: It’s About Time for Time Series Databases

#90

Post by j_s »

j_s wrote: post #59
srcmap wrote: post #57

Earlier quoted context omitted.

Just tell them SQLite is used by ALL Android / iOS phones (billions of them) to store user's critical data. I have not heard of any cases of SQLite failure in phone, or anyone needing a DB admin to recover their phone contact data.

> I have not heard of any cases of SQLite failure in phone, or anyone needing a DB admin to recover their phone contact data. For a long time (until API level 11), Android automatically silently deleted corrupted SQLite databases. https://android.stackexchange.com/questions/12388#33896 https://stackoverflow.com/questions/7764943 I learned the hard way to limit SMS storage and install Jan Berkel's SMS Backup+ https://…

Specifically (can't edit above post -- too slow):

https://issuetracker.google.com/issues/36911900#comment457

>the SQLite packaged with Android is being upgraded to version 3.6.22 in Android 2.2. I think that the SQLite database corruption bug linked in comment 451 has been fixed in Android 2.2. So _if_ that's what's causing the text message database corruption and deletion, then this bug will probably be fixed in Android 2.2

http://sqlite.1065341.n5.nabble.com/Android-database-corrupt...

>Statically link your application against SQLite 3.6.23 instead of using the SQLite 3.5.9 that is found on Android. The bug you are hitting was fixed in SQLite 3.6.2

Contrast https://web.archive.org/web/20110219041419/https://www.sqlit... (Feb 2011) vs. https://web.archive.org/web/20110729152833/https://www.sqlit... (July 2011), which could be a glitch in the Internet Archive (a partial copy of the oldest record?) but adds "Though SQLite is resistant to database corruption, it is not immune" and introduced an extensive history of known issues.

Sqlite is amazing and its development history is further testament to its current reliability, especially when used correctly!

Post reply on HN