Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How TimescaleDB compresses time-series data

roszigit.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: How TimescaleDB compresses time-series data

#21

Post by ddorian43 »

tudorg wrote: post #6

I have been working on another PG extension for timeseries ( https://github.com/xataio/deltax ) for a few months, and trying to make it score as good as possible on ClickBench. This is a project that is simply lot of fun to work on. There are many tricks that can be used to speed-up analytics, besides just type-aware compression: * for each segment you will keep things like max/min/sum, number of distinct values, blo…

What is your idea on monetizing it? Besides Timescale the other providers have found issue on monetization AFAIK. And they all restrict the S3 offloading for Enterprise accounts.

Re: How TimescaleDB compresses time-series data

#22

Post by redwood »

kevinob11 wrote: post #8
PaulWaldman wrote: post #7

Earlier quoted context omitted.

There’s an issue tracking TimescaleDB JSONB compression: https://github.com/timescale/timescaledb/issues/2978

Ha, we (my partner at our company) filed this issue 5 years ago. We have a large-ish (but not giant) json blob in one of our timescale tables that I'd love to get better compression on. It changes just frequently enough that we didn't split it into columns, but infrequently enough that it could (I think) be compressed quite nicely. Generally timescale has been great for us.

Have you tried mongodb?

Re: How TimescaleDB compresses time-series data

#23

Post by drchaim »

ahachete wrote: post #20
drchaim wrote: post #18

Is anyone running self hosted timescale? How is it going? I have a use case for it, but don’t know why I’m a bit afraid of using it.

At StackGres [1] we find Timescale to be one of the most used extensions. Timescale is quite a successful project! StackGres is actually the first solution recommended by Timescale for self-hosting with Kubernetes operators [2]. So if you are into Kubernetes (or if not, consider it, using something like K3s [3] is quite straightforward and lightweight on resources), this is probably a great option to self-host which…

I'm not into k8s things, but thanks. Still run this project with Postgres in the same machine as the app. No issues so far after 8 years. :)

Re: How TimescaleDB compresses time-series data

#24

Post by drchaim »

lkanwoqwp wrote: post #19
drchaim wrote: post #18

Is anyone running self hosted timescale? How is it going? I have a use case for it, but don’t know why I’m a bit afraid of using it.

I run it on VM the same way as a few separate PostgreSQL instances. TimescaleDB is an extension to PostgreSQL, so there's a lot of information, guides, and community support online, and Postgres is one of the most widely used databases right now. There are some things to setup but Tiger data documentation is very good.

I've tested the migration of the main table (IoT data), from 300GB to about 25GB in a non-production environment. Everything seems right, but I'm still delaying/procrastinating the prod migration, so I wanted to know how it is going for other live projects.

Re: How TimescaleDB compresses time-series data

#27

Post by ClaudiuDasca »

self_awareness wrote: post #15

Compression uses a different license than Apache. In various distros, the timescaledb installed by the package manager doesn't support other licenses than Apache. This means that there's a high chance your distro doesn't allow stock timescaledb to use compression. You probably will need to install it manually.

[dead]

Re: How TimescaleDB compresses time-series data

#28

Post by tudorg »

ddorian43 wrote: post #21
tudorg wrote: post #6

I have been working on another PG extension for timeseries ( https://github.com/xataio/deltax ) for a few months, and trying to make it score as good as possible on ClickBench. This is a project that is simply lot of fun to work on. There are many tricks that can be used to speed-up analytics, besides just type-aware compression: * for each segment you will keep things like max/min/sum, number of distinct values, blo…

What is your idea on monetizing it? Besides Timescale the other providers have found issue on monetization AFAIK. And they all restrict the S3 offloading for Enterprise accounts.

For context, I'm CTO at Xata where we offer a Postgres service that differentiates via other ways (very fast CoW branching, scale-to-zero, anonymization, etc). We often recommend people keeping production where it is (RDS, Cloud SQL, on-prem, etc.) and use Xata only for staging + branching.

We can't offer Timescale because of the license (except for the version without compression), which blocks some potential customers from using us.

So there is currently no monetization plan beyond reducing this blocker for us. An explicit goal of DeltaX is to be pure OSS, offered on Xata, and also hopefully on all other Postgres providers. Without making any hard promises for the long-term future, I will say I care deeply about it being something that reduces lock-in in the Postgres ecosystem, rather than increasing it.

Re: How TimescaleDB compresses time-series data

#29

Post by kevinob11 »

redwood wrote: post #22
kevinob11 wrote: post #8

Earlier quoted context omitted.

Ha, we (my partner at our company) filed this issue 5 years ago. We have a large-ish (but not giant) json blob in one of our timescale tables that I'd love to get better compression on. It changes just frequently enough that we didn't split it into columns, but infrequently enough that it could (I think) be compressed quite nicely. Generally timescale has been great for us.

Have you tried mongodb?

It is one of many columns. The other columns are what most of the analysis is done on.
Post reply on HN