ListenBrainz moves to TimescaleDB • Hacker News Board

Re: ListenBrainz moves to TimescaleDB

#71

Post by akulkarni » Wed, Jul 22, 2020, 7:59 PM UTC

zitterbewegung wrote: post #28

mfreed wrote: post #19

Fun fact: TimescaleDB exists because we were using InfluxDB + Postgres for a previous IoT project and also found it unworkable (developer experience, query language, reliability, scalability and performance, operations, etc). We first built TimescaleDB as "Postgres for time-series" for our own needs and then decided to open-source it for others. :-)

Have you thought of making a TimescaleDB app like the Postgres.app for macOS? Or could I use Postgres.app to make a TimescaleDB app?

Not sure if a coincidence, but someone just published this blog post today:

Installing Timescaledb on Mac OS X with Postgres.app https://prathamesh.tech/2020/07/23/installing-timescaledb-on...

Re: ListenBrainz moves to TimescaleDB

#72

Post by k-rus » Wed, Jul 22, 2020, 8:12 PM UTC

jbmsf wrote: post #62

dominotw wrote: post #58

Earlier quoted context omitted.

same way you'd insert data into postgres.

That's not really helpful. Let's assume you have a distributed system; you probably don't want all of your system components connecting directly to TimescaleDB. You also probably want to have some layer that implements queuing and handles back pressure if it can't insert into the database at the rate that events are coming in. You may want to batch insert data. I'd assume that most anyone building a system like this…

There are number of tools, which can be used in front of TimescaleDB or PostgreSQL. There are plugins for Telegraf, Kafka, Prometheus as described in https://docs.timescale.com/latest/using-timescaledb/ingestin...

Timescale builds connector and entire workflow to run Prometheus on top of TimescaleDB and support Grafana in flexible way: https://github.com/timescale/timescale-prometheus

Re: ListenBrainz moves to TimescaleDB

#73

Post by k-rus » Wed, Jul 22, 2020, 8:15 PM UTC

brightball wrote: post #47

akulkarni wrote: post #33

Earlier quoted context omitted.

TimescaleDB offers a number of backup and restore options, including wal-e (WAL-based), pg_dump & pg_restore: https://docs.timescale.com/latest/using-timescaledb/backup There are hundreds of thousands of TimescaleDB databases in production so this is generally not an issue.

Seemed like an odd issue to be outstanding. I assume if this was a real issue it would be a much bigger story.

Correct, the issue is due to confusing notices printed during pg_dump as it uses COPY TO underneath.

Re: ListenBrainz moves to TimescaleDB

#74

Post by fiddlerwoaroof » Wed, Jul 22, 2020, 8:16 PM UTC

1996 wrote: post #66

mfreed wrote: post #49

Earlier quoted context omitted.

Hi - I just want to clarify some mistake / misrepresentation about our Timescale License (TSL): 1. The TSL is not a new license, have had it in place since late 2018. What we recently announced is that multi-node TimescaleDB will be available for free under the TSL (free, source available), while (for example) clustered InfluxDB is purely proprietary (paid, closed source). 2. Our TSL license prevents offering Timesca…

> InfluxDB is purely proprietary (paid, closed source). And clickhouse is not. I just suggest skipping the timescaledb step to someone migrating from influx, and going straight to clickhouse. > For the TSL version, what it primarily restricts is the cloud providers like AWS and Azure from offering TimescaleDB-as-a-service (e.g., TimescaleDB Community on AWS RDS) If there is some kind of emergency and I need to have t…

Apart from license issues, Clickhouse is just really impressive: it’s a minor pain operationally, but in our tests it left all the Postgres-based timeseries solutions in the dust for real-time analytics without rollup tables.

Re: ListenBrainz moves to TimescaleDB

#75

Post by nwmcsween » Wed, Jul 22, 2020, 8:16 PM UTC

1996 wrote: post #38

RedShift1 wrote: post #18

I'm in the process of moving from InfluxDB to TimescaleDB myself and can't wait to get rid of the hoops I have to jump through to get InfluxDB to answer some basic questions, mostly stemming from the fact that InfluxQL doesn't support boolean expressions. Something like 'SELECT MAX(temperature) > 10 FROM...' doesn't work.

Speaking from my own experience, you may save yourself some future effort by moving directly to clickhouse. Timescale brings its own issues. If your goal is performance, you will be better served by clickhouse.

The issue I have with clickhouse is the codebase, it's an absolute behemoth and seemingly embeds musl libc? It also uses a huge amount of SIMD intrinsics for everything when SWAR or really nothing from my view looking in would have been better (memcpy, etc).

Re: ListenBrainz moves to TimescaleDB

#76

Post by gregors » Wed, Jul 22, 2020, 8:18 PM UTC

We too started off with influx but it wasn't a good fit mainly due to use having issues with high cardinality. I don't know if this is still the case with current implementations, but what it boils down to is if your data is searchable by a "user_id" really look elsewhere. That might be an oversimplification but that's the gist of it.

I was fully ready to just roll my own partitioned table and gave TimescaleDB a shot. It worked well. There was a bug we ran into, but it was an existing one documented on github and was addressed pretty quickly.

I still like influx, and would use it again but beware of the cardinality issues.

Re: ListenBrainz moves to TimescaleDB

#77

Post by jbmsf » Wed, Jul 22, 2020, 8:25 PM UTC

k-rus wrote: post #72

jbmsf wrote: post #62

Earlier quoted context omitted.

That's not really helpful. Let's assume you have a distributed system; you probably don't want all of your system components connecting directly to TimescaleDB. You also probably want to have some layer that implements queuing and handles back pressure if it can't insert into the database at the rate that events are coming in. You may want to batch insert data. I'd assume that most anyone building a system like this…

There are number of tools, which can be used in front of TimescaleDB or PostgreSQL. There are plugins for Telegraf, Kafka, Prometheus as described in https://docs.timescale.com/latest/using-timescaledb/ingestin... Timescale builds connector and entire workflow to run Prometheus on top of TimescaleDB and support Grafana in flexible way: https://github.com/timescale/timescale-prometheus

Perfect. Thank you for the reference.

Re: ListenBrainz moves to TimescaleDB

#78

Post by decafninja » Wed, Jul 22, 2020, 8:41 PM UTC

As someone who wants to pick up a time series DB to learn, what would be the best in terms of being the "industry standard"? InfluxDB? TimescaleDB?

I'm familiar with some basics of kdb and use it often in my day job, but from what I understand that isn't widely used outside of finance?

Re: ListenBrainz moves to TimescaleDB

#79

Post by rweichler » Wed, Jul 22, 2020, 8:45 PM UTC

Figured I'd use this as an opportunity to plug my own service: https://eqe.fm

Only works on jailbroken devices but it works well, has a local backup, and has been maintained (by me) for 2 years now.

Server costs are $2.50/mo, so this will stay up as long as I am alive.

Re: ListenBrainz moves to TimescaleDB

#80

Post by akulkarni » Wed, Jul 22, 2020, 8:49 PM UTC

decafninja wrote: post #78

As someone who wants to pick up a time series DB to learn, what would be the best in terms of being the "industry standard"? InfluxDB? TimescaleDB? I'm familiar with some basics of kdb and use it often in my day job, but from what I understand that isn't widely used outside of finance?

(I work at TimescaleDB.)

If you are familiar with Postgres and/or SQL, then you may want to start with TimescaleDB. It's just Postgres for time-series. Full SQL, so it's possible to be productive instantly.