Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ListenBrainz moves to TimescaleDB

blog.metabrainz.org

Post reply on HN
Print view View original

41–50 of 100 posts

Re: ListenBrainz moves to TimescaleDB

#41

Post by 1996 »

RedShift1 wrote: post #39
1996 wrote: post #38

Earlier quoted context omitted.

Speaking from my own experience, you may save yourself some future effort by moving directly to clickhouse. Timescale brings its own issues. If your goal is performance, you will be better served by clickhouse.

Can you elaborate on those issues?

It would be too long. To quickly summarize, from the pain of backups (unless you setup a WAL replica, the load may take your database down), the large size of the data on disk (timescale does offer some compression now, but it's still too much), the low performance of large queries, the memory requirements - it's death by a thousand papercuts!

Don't get me wrong, timescale is a great way to get started with time series - just like sqlite is a great way to get started with databases if all you know is nosql.

However, it quickly brings its own challenges - and the new license is the cherry on the cake: it is locking you down to your own infrastructure unless you want to pay for timescale own SAAS offering (and then prey they do not alter the condition of the deal too much later)

It is just not worth it, unless you have a very small problem, or you can afford to have people concentrating on timescale maintenance - and in this case, you would be getting better bang for bucks by having these people work on clickhouse.

I'm speaking only from my own experience. I have relatively large servers dedicated to time series (about 100T of disk space, between 128 and 256 Gb of RAM). They were going to be retired for even bigger servers. Instead, we experimented with clickhouse on one of the recently decommissioned servers. We could not believe the benchmarks! Moving to clickhouse has improved the performance on about every metric. Yes, it required some minor SQL rewrites, about 1 day of work total, but unless your hardware is free and your queries are set in stone, clickhouse makes more sense.

Re: ListenBrainz moves to TimescaleDB

#42

Post by 120bits »

Interesting read and thanks for sharing.

Not too long ago, I was asked to work on some analytics project and it required time-series data. I'm not a rockstar programmer and don't really know much about trends. So, I ended up googling and stumble upon InfluxDB. It felt like that right choice and I started playing with it. As the time passed, I realized that it might be a good software and I'm sure people love InfluxDB, but it wasn't the right choice for me. I didn't really like the docs, maybe its good now. And I had the same feeling about query syntax, it felt weird.

I moved to TimescaleDB and never looked back. I have it production for almost 2 months now. 20 tables and over 100Million writes/week. One of things I really liked was staging, I don't use docker and or anything fancy. I have bash script that and it runs on centos box and all timescale extension and postgres database are packaged together.

I was impressed by the timescale compression feature. I wasn't using it earlier because I had to be careful about what columns I need to segmentby. I would love to see some more features but I'm sure timescaledb team is already on it.

Re: ListenBrainz moves to TimescaleDB

#43

Post by sgt »

sciurus wrote: post #32
sgt wrote: post #8

Earlier quoted context omitted.

You can't for example delete data from a time period to another, to the best of my knowledge.

This is actually pretty straightforward. For example, `DELETE FROM "foo" WHERE time >= now() - 2d AND time < now() - 1d`

Ok, we had an issue deleting data from a series. The delete just hung and never completed. I assumed this was not really possible on Influx. Perhaps this is an edge case.

Re: ListenBrainz moves to TimescaleDB

#45

Post by akulkarni »

justinclift wrote: post #35
akulkarni wrote: post #27

Earlier quoted context omitted.

TimescaleDB definitely supports backups :-) Here is a page from our docs on how to perform Backup & Restore: https://docs.timescale.com/latest/using-timescaledb/backup Not sure what's going on in that one Github issue, but we are looking into it.

It seems to be affecting multiple people too. :(

That issue is now closed by the original author:

"Data is successfully dumped. also i can see the constraints, indexes are also copied successfully."

https://github.com/timescale/timescaledb/issues/1835

Re: ListenBrainz moves to TimescaleDB

#46

Post by akulkarni »

thejosh wrote: post #23

I really want to love timescaledb, it's great.. except for the minor issue of not being able to back up. https://github.com/timescale/timescaledb/issues/1835

Posted elsewhere, but also posting here for posterity:

That issue is now closed by the original author:

"Data is successfully dumped. also i can see the constraints, indexes are also copied successfully."

https://github.com/timescale/timescaledb/issues/1835

Re: ListenBrainz moves to TimescaleDB

#47

Post by brightball »

akulkarni wrote: post #33
brightball wrote: post #31

Earlier quoted context omitted.

That seems like a pretty big deal. Does a WAL backup approach work?

TimescaleDB offers a number of backup and restore options, including wal-e (WAL-based), pg_dump & pg_restore: https://docs.timescale.com/latest/using-timescaledb/backup There are hundreds of thousands of TimescaleDB databases in production so this is generally not an issue.

Seemed like an odd issue to be outstanding. I assume if this was a real issue it would be a much bigger story.

Re: ListenBrainz moves to TimescaleDB

#48

Post by cevian »

1996 wrote: post #41
RedShift1 wrote: post #39

Earlier quoted context omitted.

Can you elaborate on those issues?

It would be too long. To quickly summarize, from the pain of backups (unless you setup a WAL replica, the load may take your database down), the large size of the data on disk (timescale does offer some compression now, but it's still too much), the low performance of large queries, the memory requirements - it's death by a thousand papercuts! Don't get me wrong, timescale is a great way to get started with time seri…

(TimescaleDB engineer here) Some of the comments here sound technically off.

We've never seen a backup take down a machine. The backups we use are the same as Postgres which are used by millions of companies without a problem (and can be streaming incremental backups like pgBackrest, WAL-E, etc. or whole-database backups like pg_dump). As with any DB you do have to size and configure your database correctly (which these days isn't hard).

We've never seen anybody claim that ClickHouse offers significantly better compression than we do overall. Obviously compression depends heavily on data distribution and I'm sure you could make up a dataset where clickhouse does better (just as you could where TimescaleDB does better). But on real distributions we don't see this at all, we do pretty advanced columnar compression on a per-datatype basis [1], and see median space reduction of 95% from compression across users.

Large queries is a weird claim to make since Postgres has more different types of indexes than Clickhouse and has support for multiple indexes. If you are processing all of your data for all your queries then yes, click house sequential scans may be better. But that's less common, and also where TimescaleDB continuous aggregates come in.

We've seen customers successfully use our single-node version with 100s of billions of rows so claiming that we are just for small use-cases is simply untrue, and especially with the launch of multi-node TimescaleDB.

I understand people may have different preferences and experiences, but some of these felt a bit off to me.

[1] https://blog.timescale.com/blog/building-columnar-compressio...

Re: ListenBrainz moves to TimescaleDB

#49

Post by mfreed »

1996 wrote: post #41
RedShift1 wrote: post #39

Earlier quoted context omitted.

Can you elaborate on those issues?

It would be too long. To quickly summarize, from the pain of backups (unless you setup a WAL replica, the load may take your database down), the large size of the data on disk (timescale does offer some compression now, but it's still too much), the low performance of large queries, the memory requirements - it's death by a thousand papercuts! Don't get me wrong, timescale is a great way to get started with time seri…

Hi - I just want to clarify some mistake / misrepresentation about our Timescale License (TSL):

1. The TSL is not a new license, have had it in place since late 2018. What we recently announced is that multi-node TimescaleDB will be available for free under the TSL (free, source available), while (for example) clustered InfluxDB is purely proprietary (paid, closed source).

2. Our TSL license prevents offering TimescaleDB-as-a-service, it absolutely does NOT prevent you from running/offering a SaaS service or from utilizing cloud services/infra (you say "it is locking you down to your own infrastructure unless you want to pay for timescale own SAAS offering"). Specifically, Timescale offers a pure Apache-2 version and a "Timescale License" (TSL) Community version. For the TSL version, what it primarily restricts is the cloud providers like AWS and Azure from offering TimescaleDB-as-a-service (e.g., TimescaleDB Community on AWS RDS). Many thousands of companies use our community version for free to build SaaS services running on their own AWS instances.

Re: ListenBrainz moves to TimescaleDB

#50

Post by mfreed »

120bits wrote: post #42

Interesting read and thanks for sharing. Not too long ago, I was asked to work on some analytics project and it required time-series data. I'm not a rockstar programmer and don't really know much about trends. So, I ended up googling and stumble upon InfluxDB. It felt like that right choice and I started playing with it. As the time passed, I realized that it might be a good software and I'm sure people love InfluxDB…

Hey 120bits - thanks for the nice words!

What new/other features would you like to see? (Also feel free to join slack.timescale.com or reach out at mike (at) timescale.com)

Post reply on HN