> Back in 2014...developers...could use relational databases with SQL interfaces, which are easy to use but they don’t scale well. I'm skeptical.
Guessing they meant to write "... for time series aggregation"
It’s About Time for Time Series Databases
51–60 of 151 posts
Re: It’s About Time for Time Series Databases
#52Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast.
But try telling people you're using SQLite to store critical data...
Re: It’s About Time for Time Series Databases
#53How about one made by Rich Hickey — https://www.datomic.com/
is Datomic good for timeseries ? AFAIWA it's good for rare-write often-read applications
Re: It’s About Time for Time Series Databases
#54The article recognizes that several time series databases already exist. They also say, "we aren't trying to compete against kdb+." They explain how they can handle time series data better than NoSQL databases that aren't time series focused. But what are they doing better than the existing time series databases? Surely they must have some advantage or they wouldn't have raised 16.1 million dollars.
Every time I read about some new solution to storing time series data, I always feel like I must be doing something wrong, but I've run into _zero_ problems yet.
Every time I have to store time series data, I never really need ACID transactions. I definitely don't ever need to do updates or upserts. It's always write-once-read-many. ElasticSearch has always been the obvious choice to me and it has worked extremely well. Information retrieval is incredibly robust and for times that I'm worried about consistency, I use Postgres's JSON capabilities and write first to there. You can have your application sanity-check between the two if you're worried about ElasticSearch not receiving/losing data.
I find it really hard to beat.
Re: It’s About Time for Time Series Databases
#55Earlier quoted context omitted.
> But is anyone else concerned about using a DB solely from a company built specifically around that DB? It worked out pretty well for Oracle. (SCNR)
I'd almost rather be locked in to a cheap, may-deprecate DB than a really expensive one. The former at least forces you to move, the latter just costs your business a lot especially when non-doers run the tech show and will sink money over re-impl every time (as opposed to making measured decisions).
FWIW, I found a table with prices for Oracle database licenses once, and it made Microsoft SQL Server look cheap. I am not sure if that means MSSQL is cheap, though. At least I am fairly confident that Microsoft is not going out of business anytime soon.
In my current job, I needed to get intimately familiar with the database our ERP software uses, because my boss needed lots of reports that the ERP software just did not do (unless paid an expensive consultant to customize the software for you). And the idea of having to move all that data to a different database makes me shiver in horror. I can understand why a business would be very reluctant to move their data from one database (both schema and engine) to another.
For business-critical tasks, I prefer a boring, reliable solution. But it is good to know that there are still interesting developments going on in the database area, building tomorrow's boring, reliable solutions.
Re: It’s About Time for Time Series Databases
#56Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
What's your replication/HA story?
Re: It’s About Time for Time Series Databases
#57Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
Just tell them SQLite is used by ALL Android / iOS phones (billions of them) to store user's critical data.
I have not heard of any cases of SQLite failure in phone, or anyone needing a DB admin to recover their phone contact data.
Re: It’s About Time for Time Series Databases
#58Earlier quoted context omitted.
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
What's your replication/HA story?
Yes, collecting/aggregating for querying seems like the sticking point.
Re: It’s About Time for Time Series Databases
#59Earlier quoted context omitted.
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
Just tell them SQLite is used by ALL Android / iOS phones (billions of them) to store user's critical data. I have not heard of any cases of SQLite failure in phone, or anyone needing a DB admin to recover their phone contact data.
> I have not heard of any cases of SQLite failure in phone, or anyone needing a DB admin to recover their phone contact data.
For a long time (until API level 11), Android automatically silently deleted corrupted SQLite databases. https://android.stackexchange.com/questions/12388#33896 https://stackoverflow.com/questions/7764943
I learned the hard way to limit SMS storage and install Jan Berkel's SMS Backup+ https://github.com/jberkel/sms-backup-plus/#readme after losing my first Android SMS database (this may or may not have been SQLite's fault).
Re: It’s About Time for Time Series Databases
#60Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…
We use a combination of SQLite and a sharding frontend service. One SQLite database file per device, one table per sensor, table contents are timestamp and measured value. As simple as it gets, easy to scale, and damn fast. But try telling people you're using SQLite to store critical data...
Wow, that's a pretty neat approach. I have a bunch of questions, if you don't mind..
Do you do filesystem replication or something else for HA? Is this running in one of the cloud cloud IaaS providers or something else?
Why did you go with this approach? Eg: Was it a deliberate decision after evaluating other options, a "this is what we know" type situation, or did it grow organically from something smaller?
How long has it been running? Whats the rough size of data you have (GB/TB, number of servers, number of DB instances)? Any reliability problems?
If you were building this service from scratch, would you do it this way again?