Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s About Time for Time Series Databases

nextplatform.com

Post reply on HN
Print view View original

101–110 of 151 posts

Re: It’s About Time for Time Series Databases

#101

Post by matte_black »

stdbrouw wrote: post #24
teej wrote: post #18

Earlier quoted context omitted.

This simply doesn’t work when you have sparsely populated dimensions and/or you don’t know what dimensions are important in advance. Both of these are very common. That’s why you don’t see a higher prevalence of estimated measurement.

> you don’t know what dimensions are important in advance But again this is a huge red flag. I've seen so many data science projects that started with "well, let's just get started with collecting everything and we will figure out what is important later on" and then spent so much time on infrastructure that no useful insights were ever produced.

Unfortunately there’s no way to know if useful insights won’t be produced unless you explore the full data set.

Re: It’s About Time for Time Series Databases

#102

Post by da_chicken »

wenc wrote: post #85
baldfat wrote: post #69

Earlier quoted context omitted.

> But try telling people you're using SQLite to store critical data... SQLite is solid and trusted by many but for some unknown reason some people refuse to see it as a solution for their needs. I have seen many databases that could easily been replaced by SQLite.

I looked into SQLite a while ago, and it wasn't so much its robustness that was the deal-breaker, but more the fact that it couldn't handle writes from parallel processes from multiple users (i.e. concurrency) like a normal SQL database could. I see SQLite as a file-format with a SQL interface.

Yep. Parallel performance is the biggest issue I've seen with SQLite.

SpiceWorks is the biggest example of a product that doesn't scale because they've elected to use SQLite (and only SQLite). There's no hard limit on the number of devices, but most people say not to exceed about 1,000 devices. We tested it out against 800 devices. We started it on Friday. On Monday, it was still running.

The only other issue I've seen with it is with the data typing. Type inference is much weaker than strict typing, and sometimes it doesn't quite store things the way you'd expect. Granted, the last time I saw problems with that was years ago, but I've seen it.

Re: It’s About Time for Time Series Databases

#103

Post by creeble »

eloff wrote: post #73
creeble wrote: post #62

Earlier quoted context omitted.

Simpler than a log file? What's the significant advantage of SQLite vs just appending to a file?

Transactions? Crash data safety? SQL queries? Portability?

I don't see how any of those apply for a two-column 'db' of timestamp and data.

Re: It’s About Time for Time Series Databases

#104

Post by jnordwick »

akulkarni wrote: post #80
olympus wrote: post #47

The article recognizes that several time series databases already exist. They also say, "we aren't trying to compete against kdb+." They explain how they can handle time series data better than NoSQL databases that aren't time series focused. But what are they doing better than the existing time series databases? Surely they must have some advantage or they wouldn't have raised 16.1 million dollars.

A few reasons why you may want to use TimescaleDB vs other time series DBs: 1. For some developers, just having a SQL interface to time-series data (while maintaining insert/query performance at scale) is good enough reason to use TimescaleDB. For example, when trying to express complex queries, or when trying to connect to SQL-based visualization tools (e.g., Tableau), or anything else in the PostgreSQL ecosystem. 2…

I hate to be "that one" but:

> There was no database that did all of this when we decided to build Timescale. (If one did exist, we would have used it).

If you removed the based on Postgres part, except for KDB, IQ, Vertica, and a few others. I can definitely see a price argument though (i.e, the same but cheaper) as those all tend to be a expensive.

Re: It’s About Time for Time Series Databases

#105

Post by Johnny555 »

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

There's also snowflakedb which was in the news today after getting a pile of funding. Though it seems to be cloud-only, not something you can run yourself.

Re: It’s About Time for Time Series Databases

#106

Post by lbruder »

gregmac wrote: post #60
lbruder wrote: post #52

Earlier quoted context omitted.

We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...

Wow, that's a pretty neat approach. I have a bunch of questions, if you don't mind.. Do you do filesystem replication or something else for HA? Is this running in one of the cloud cloud IaaS providers or something else? Why did you go with this approach? Eg: Was it a deliberate decision after evaluating other options, a "this is what we know" type situation, or did it grow organically from something smaller? How long…

Don't mind at all. ATM we're using a simple VMWare server backing up the virtual HDD every night, with a dirty hack thrown in (no full backups on weekends as that's when the bigger jobs run, so I dump the current day to a CSV on a smaller virtual HDD that's still saved daily). We're running on our own systems as our customers are pretty paranoid about where their data is sent.

The history of this system is pretty organic, lots of experimentation. Started with Postgres, but that was too slow if a select statement returns tens of thousands of rows. MariaDB was way faster but became erratic once we passed about 2 billions of records. Now with SQLite we are at about 180 GB, about three years of data so far, so not really "big data" territory yet.

Constantly looking for better alternatives. InfluxDB looks promising, but maybe we'll switch to a "hot/cold" approach instead, keeping whole "cold" days as JSON objects in JSONB fields in Postgres again while storing "hot" data in SQLite. We'll see...

So far the system has been rock solid. In the last few weeks we went live for real, now storing about 2k records a second. Not a problem so far.

Re: It’s About Time for Time Series Databases

#107

Post by andrew_deane__ »

j_s wrote: post #45
siculars wrote: post #25

Earlier quoted context omitted.

Riak the database survives Basho the company. All assets (IP) were bought by bet365 (A large Basho customer) and made open source. Development continues. /former Basho employee.

> made open source I hope things work out at least as well as they have for RethinkDB's resurrection/transition! The Riak Users mailing list seems like the best place to follow along: http://riak-users.197444.n3.nabble.com https://github.com/basho/riak_kv/tree/develop-2.2.5 (November 2017) http://bet365techblog.com/riak-workshop-summary (October 2017) > the immediate goal of releasing a known good build as soon as po…

All IP has been made open by bet365 (including RIAK TS) and development is continuing, both in the community and within the company. bet365 are actively recruiting to support this effort, and are committed to the long term future of RIAK. Latest release is to be cut Jan / Feb with the help of the community.

Re: It’s About Time for Time Series Databases

#108

Post by baldfat »

da_chicken wrote: post #102
wenc wrote: post #85

Earlier quoted context omitted.

I looked into SQLite a while ago, and it wasn't so much its robustness that was the deal-breaker, but more the fact that it couldn't handle writes from parallel processes from multiple users (i.e. concurrency) like a normal SQL database could. I see SQLite as a file-format with a SQL interface.

Yep. Parallel performance is the biggest issue I've seen with SQLite. SpiceWorks is the biggest example of a product that doesn't scale because they've elected to use SQLite (and only SQLite). There's no hard limit on the number of devices, but most people say not to exceed about 1,000 devices. We tested it out against 800 devices. We started it on Friday. On Monday, it was still running. The only other issue I've se…

There are plenty of problems that don't need parallel performance.

Re: It’s About Time for Time Series Databases

#109

Post by jmkirby »

creeble wrote: post #103
eloff wrote: post #73

Earlier quoted context omitted.

Transactions? Crash data safety? SQL queries? Portability?

I don't see how any of those apply for a two-column 'db' of timestamp and data.

Except.. compliance staff who want to use some ahem wonderful standards complaint reporting tool.

There's just one solitary mention of Informix in the whole discussion, as I'm typing this. And that's a comment lumping Informix with everything else commercial and expensive.

I thought Informix owned the embedded dB space at a certain rollout complexity. Remember the backups... Informix speaks json too, including at some quite low level primitives. Whether you buy a product like Informix Edge plus tools or not, depends on what is sending the time series data.

Post reply on HN